Selective processing of sensor data

ABSTRACT

Systems and methods for navigating a vehicle within an environment are provided. In one aspect, a method comprises: (a) selecting, with aid of a processor, a subset of a plurality of sensors to be used for navigating the vehicle within the environment based on one or more predetermined criteria, wherein the plurality of sensors are arranged on the vehicle such that each sensor of the plurality of sensors is configured to obtain sensor data from a different field of view; (b) processing, with aid of the processor, the sensor data from the selected sensor(s) so as to generate navigation information for navigating the vehicle within the environment; and (c) outputting, with aid of the processor, signals for controlling the vehicle based on the navigation information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of InternationalApplication Serial No. PCT/CN2014/095983, filed Dec. 31, 2014, which isincorporated herein by reference in its entirety.

BACKGROUND

Unmanned vehicles such as unmanned aerial vehicles (UAVs) can be usedfor performing surveillance, reconnaissance, and exploration tasks in awide variety of environments for military and civilian applications. AUAV may be manually controlled by a remote user, or may operate in asemi-autonomous or fully autonomous manner. Such UAVs can includesensors configured to collect sensor data from the surroundingenvironment and processors to calculate and determine information fornavigating the UAV from the collected data.

Existing approaches for assessing navigation information may be lessthan optimal in some instances. For example, the sensor data gatheredfrom a single sensor may be of poor quality. On the other hand, ifsensor data from multiple sensors are processed to assess or determinestate information, the processing delay of the system may be increased.Sensor data of poor quality or increased processing delays may have adetrimental effect on UAV functions.

SUMMARY

Embodiments disclosed herein provide improved approaches for assessingstate information of vehicles such as UAVs within an environment. Inmany embodiments, a UAV includes a plurality of imaging devices used tocollect information regarding the surrounding environment. The dataobtained from each of the imaging devices can be processed, e.g., todetermine the quality of the image data over a time interval for eachimaging device. Based on the quality of the image data, a subset of theimaging device's image data can be selected for processing in order toassess state information of the UAV. Advantageously, the selectiveprocessing approaches described herein provide improved accuracy of theassessed state information while conserving computing resources, and canbe used to improve UAV functionality in diverse environment types andoperating conditions.

Thus, in one aspect, a navigation system is provided. The systemcomprises a vehicle comprising a plurality of imaging devices eachconfigured to capture a plurality of images; and one or more processorsoperably coupled to the plurality of imaging devices and individually orcollectively configured to: (a) calculate a feature point number in eachimage of the plurality of images from each imaging device; (b) select atleast one of the plurality of imaging devices based on said featurepoint number; and (c) assess state information for the vehicle using theplurality of images from the selected imaging device(s).

In some embodiments, the vehicle can be an unmanned aerial vehicle. Theunmanned aerial vehicle may weigh no more than 10 kg. The maximumdimension of the unmanned aerial vehicle may be no more than 1.5 m. Theunmanned aerial vehicle can be configured to fly at a height of no morethan 400 m. Optionally, the unmanned aerial vehicle can be configured todetect the presence of a restricted flight region and not fly within apredetermined distance of the restricted flight region. The restrictedflight region may be an airport. The unmanned aerial vehicle can be amulti-rotor aircraft.

In some embodiments, the vehicle can comprise one or more propulsionunits configured to effect movement of the vehicle. The one or morepropulsion units can comprise one or more rotors configured to providelift to the vehicle.

In some embodiments, the plurality of imaging devices are arranged onthe vehicle such that each imaging device of the plurality of imagingdevices is configured to capture the plurality of images from adifferent field of view. In some embodiments, the plurality of imagingdevices can comprise at least three imaging devices. Alternatively, theplurality of imaging devices can comprise at least four imaging devices.The plurality of imaging devices can be each oriented in a differentdirection relative to the vehicle. The different directions can beorthogonal directions. Alternatively, or in combination, the differentdirections can comprise at least four different directions. At least oneof the different directions can be substantially along a direction ofmotion of the vehicle. The plurality of imaging devices can be locatedat three or more of the following locations: a front side, rear side,left side, right side, upper side, or lower side of the vehicle.

In some embodiments, the plurality of images can comprise a plurality ofsuccessive image frames captured over a predetermined time interval. Thepredetermined time interval can be within a range from about 0.02seconds to about 0.1 seconds.

In some embodiments, the feature point number in each image can beindicative of saliency of said image. The feature point number in eachimage can be calculated using a corner detection algorithm. The cornerdetection algorithm can be a Features from Accelerated Segment Test(FAST) algorithm.

In some embodiments, step (b) can comprise assessing whether saidfeature point number exceeds a predetermined threshold.

In some embodiments, the state information can comprise at least one ofa position, an attitude, a velocity, or an acceleration of the vehicle.The attitude can comprise at least one of a roll orientation, a pitchorientation, or a yaw orientation of the vehicle.

In some embodiments, the one or more processors can further beconfigured to: (d) output control signals for effecting movement of thevehicle based on the state information.

In some embodiments, the steps (a)-(c) can be repeated during operationof the vehicle. The steps (a)-(c) can be repeated about once every 0.02seconds to about once every 0.1 seconds.

In another aspect, a method for assessing state information of a movingvehicle attached to a plurality of imaging devices is provided. Themethod comprises: (a) capturing a plurality of images with each imagingdevice of the plurality of imaging devices; (b) calculating, with aid ofa processor, a feature point number in each image of the plurality ofimages from each imaging device; (c) selecting, with aid of theprocessor, at least one of the plurality of imaging devices based onsaid feature point number; and (d) assessing, with aid of the processor,state information for the vehicle using the plurality of images from theselected imaging device(s).

In some embodiments, the vehicle can be an unmanned aerial vehicle. Theunmanned aerial vehicle may weigh no more than 10 kg. The maximumdimension of the unmanned aerial vehicle may be no more than 1.5 m. Theunmanned aerial vehicle can be configured to fly at a height of no morethan 400 m. Optionally, the unmanned aerial vehicle can be configured todetect the presence of a restricted flight region and not fly within apredetermined distance of the restricted flight region. The restrictedflight region may be an airport. The unmanned aerial vehicle can be amulti-rotor aircraft.

In some embodiments, the vehicle can comprise one or more propulsionunits configured to effect movement of the vehicle. The one or morepropulsion units can comprise one or more rotors configured to providelift to the vehicle.

In some embodiments, the plurality of imaging devices are arranged onthe vehicle such that each of the plurality of images is captured from adifferent field of view. In some embodiments, the plurality of imagingdevices can comprise at least three imaging devices. Alternatively, theplurality of imaging devices can comprise at least four imaging devices.The plurality of imaging devices can be each oriented in a differentdirection relative to the vehicle. The different directions can beorthogonal directions. Alternatively, or in combination, the differentdirections can comprise at least four different directions. At least oneof the different directions can be substantially along a direction ofmotion of the vehicle. The plurality of imaging devices can be locatedat three or more of the following locations: a front side, rear side,left side, right side, upper side, or lower side of the vehicle.

In some embodiments, the plurality of images can comprise a plurality ofsuccessive image frames captured over a predetermined time interval. Thepredetermined time interval can be within a range from about 0.02seconds to about 0.1 seconds.

In some embodiments, the feature point number in each image can beindicative of saliency of said image. The feature point number in eachimage can be calculated using a corner detection algorithm. The cornerdetection algorithm can be a Features from Accelerated Segment Test(FAST) algorithm.

In some embodiments, step (c) can comprise assessing whether saidfeature point number exceeds a predetermined threshold.

In some embodiments, the state information can comprise at least one ofa position, an attitude, a velocity, or an acceleration of the vehicle.The attitude can comprise at least one of a roll orientation, a pitchorientation, or a yaw orientation of the vehicle.

In some embodiments, the method can further comprise outputting controlsignals for effecting movement of the vehicle based on the stateinformation.

In some embodiments, the steps (a)-(d) can be repeated during operationof the vehicle. The steps (a)-(d) can be repeated about once every 0.02seconds to about once every 0.1 seconds.

In another aspect, a navigation system is provided. The systemcomprises: a vehicle comprising one or more propulsion units configuredto effect movement of the vehicle and a plurality of imaging eachconfigured to capture a plurality of images; and one or more processorsoperably coupled to the plurality of imaging devices and individually orcollectively configured to: (a) assess image quality of the plurality ofimages from each imaging device; (b) select at least one of theplurality of imaging devices based on the assessment of step (a); and(c) assess state information for the vehicle using the plurality ofimages from the selected imaging device(s). In some embodiments, the oneor more processors are further configured to: (d) output control signalsto the one or more propulsion units for effecting the movement of thevehicle, based on the state information.

In some embodiments, the vehicle can be an unmanned aerial vehicle. Theunmanned aerial vehicle may weigh no more than 10 kg. The maximumdimension of the unmanned aerial vehicle may be no more than 1.5 m. Theunmanned aerial vehicle can be configured to fly at a height of no morethan 400 m. Optionally, the unmanned aerial vehicle can be configured todetect the presence of a restricted flight region and not fly within apredetermined distance of the restricted flight region. The restrictedflight region may be an airport. The unmanned aerial vehicle can be amulti-rotor aircraft.

In some embodiments, the one or more propulsion units can comprise oneor more rotors configured to provide lift to the vehicle.

In some embodiments, the plurality of imaging devices are arranged onthe vehicle such that each imaging device of the plurality of imagingdevices is configured to capture the plurality of images from adifferent field of view. In some embodiments, the plurality of imagingdevices can comprise at least three imaging devices. Alternatively, theplurality of imaging devices can comprise at least four imaging devices.The plurality of imaging devices can be each oriented in a differentdirection relative to the vehicle. The different directions can beorthogonal directions. Alternatively or in combination, the differentdirections can comprise at least four different directions. At least oneof the different directions can be substantially along a direction ofmotion of the vehicle. The plurality of imaging devices can be locatedat three or more of the following locations: a front side, rear side,left side, right side, upper side, or lower side of the vehicle.

In some embodiments, the plurality of images can comprise a plurality ofsuccessive image frames captured over a predetermined time interval. Thepredetermined time interval can be within a range from about 0.02seconds to about 0.1 seconds.

In some embodiments, the image quality can be based on a feature pointnumber in each image of said plurality of images. The feature pointnumber in each image can be calculated using a corner detectionalgorithm. The corner detection algorithm can be a Features fromAccelerated Segmented Test (FAST) algorithm. The image quality can bebased on saliency of each image of said plurality of images. The imagequality can be based on at least one of an exposure level or contrastlevel in each image of said plurality of images. The image quality canbe based on suitability of said plurality of images for use in assessingthe state information for the vehicle.

In some embodiments, step (a) can comprise assessing whether the imagequality of said plurality of images exceeds a predetermined threshold.Step (a) can comprise identifying which of the plurality of images hasthe highest image quality.

In some embodiments, the state information can comprise at least one ofa position, an attitude, a velocity, or an acceleration of the vehicle.The attitude comprises at least one of a roll orientation, a pitchorientation, or a yaw orientation of the vehicle.

In some embodiments, steps (a)-(c) can be repeated during operation ofthe vehicle. The steps (a)-(c) can be repeated about once every 0.02seconds to about once every 0.1 seconds.

In another aspect, a method for controlling a moving vehicle attached toa plurality of imaging devices is provided. The method comprises: (a)capturing a plurality of images with each imaging device of theplurality of imaging devices; (b) assessing, with aid of a processor,image quality of the plurality of images from each imaging device; (c)selecting, with aid of the processor, at least one of the plurality ofimaging devices based on the assessment of step (b); and (d) assessing,with aid of the processor, state information for the vehicle using theplurality of images from the selected imaging device(s). In someembodiments, the method further comprises: (e) outputting, with aid ofthe processor, control signals to one or more propulsion units mountedon the vehicle for effecting movement of the vehicle, based on the stateinformation.

In some embodiments, the vehicle can be an unmanned aerial vehicle. Theunmanned aerial vehicle may weigh no more than 10 kg. The maximumdimension of the unmanned aerial vehicle may be no more than 1.5 m. Theunmanned aerial vehicle can be configured to fly at a height of no morethan 400 m. Optionally, the unmanned aerial vehicle can be configured todetect the presence of a restricted flight region and not fly within apredetermined distance of the restricted flight region. The restrictedflight region may be an airport. The unmanned aerial vehicle can be amulti-rotor aircraft.

In some embodiments, the one or more propulsion units can comprise oneor more rotors configured to provide lift to the vehicle.

In some embodiments, the plurality of imaging devices are arranged onthe vehicle such that each of the plurality of images is captured from adifferent field of view. In some embodiments, the plurality of imagingdevices can comprise at least three imaging devices. Alternatively, theplurality of imaging devices can comprise at least four imaging devices.

In some embodiments, the plurality of imaging devices can be eachoriented in a different direction relative to the vehicle. The differentdirections can be orthogonal directions. Alternatively, or incombination, the different directions can comprise at least fourdifferent directions. At least one of the different directions can besubstantially along a direction of motion of the vehicle. The pluralityof imaging devices can be located at three or more of the followinglocations: a front side, rear side, left side, right side, upper side,or lower side of the vehicle.

In some embodiments, the plurality of images can comprise a plurality ofsuccessive image frames captured over a predetermined time interval. Thepredetermined time interval can be within a range from about 0.02seconds to about 0.1 seconds.

In some embodiments, the image quality can be based on a feature pointnumber in each image of said plurality of images. The feature pointnumber in each image can be calculated using a corner detectionalgorithm. The corner detection algorithm can be a Features fromAccelerated Segmented Test (FAST) algorithm. The image quality can bebased on saliency of each image of said plurality of images. The imagequality can be based on at least one of an exposure level or contrastlevel in each image of said plurality of images. The image quality canbe based on suitability of said plurality of images for use in assessingthe state information for the vehicle.

In some embodiments, step (b) can comprise assessing whether the imagequality of said plurality of images exceeds a predetermined threshold.

In some embodiments, step (b) can comprise identifying which of theplurality of images has the highest image quality.

In some embodiments, the state information can comprise at least one ofa position, an attitude, a velocity, or an acceleration of the vehicle.The attitude comprises at least one of a roll orientation, a pitchorientation, or a yaw orientation of the vehicle.

In some embodiments, steps (a)-(d) can be repeated during operation ofthe vehicle. The steps (a)-(d) can be repeated about once every 0.02seconds to about once every 0.1 seconds.

In another aspect, a navigation system is provided. The systemcomprises: a vehicle comprising a plurality of imaging devices eachconfigured to capture a plurality of images, wherein the plurality ofimaging devices comprise a primary imaging device and one or moresecondary imaging devices; and one or more processors operably coupledto the plurality of imaging devices and individually or collectivelyconfigured to: (a) assess image quality of the plurality of images fromthe primary imaging device to determine whether said image quality meetsa predetermined threshold; (b) assess image quality of the plurality ofimages from the one or more secondary imaging devices if the imagequality of step (a) does not meet the predetermined threshold; (c)select at least one of the one or more secondary imaging devices basedon the assessment of step (b); and (d) assess state information for thevehicle using the plurality of images from the selected secondaryimaging device(s).

In some embodiments, the vehicle can be an unmanned aerial vehicle. Theunmanned aerial vehicle may weigh no more than 10 kg. The maximumdimension of the unmanned aerial vehicle may be no more than 1.5 m. Theunmanned aerial vehicle can be configured to fly at a height of no morethan 400 m. Optionally, the unmanned aerial vehicle can be configured todetect the presence of a restricted flight region and not fly within apredetermined distance of the restricted flight region. The restrictedflight region may be an airport. The unmanned aerial vehicle can be amulti-rotor aircraft.

In some embodiments, the one or more propulsion units can comprise oneor more rotors configured to provide lift to the vehicle.

In some embodiments, the plurality of imaging devices are arranged onthe vehicle such that each imaging device of the plurality of imagingdevices is configured to capture the plurality of images from adifferent field of view. In some embodiments, the plurality of imagingdevices can comprise at least three imaging devices. Alternatively, theplurality of imaging devices can comprise at least four imaging devices.The plurality of imaging devices can be each oriented in a differentdirection relative to the vehicle. The different directions can beorthogonal directions. Alternatively or in combination, the differentdirections can comprise at least four different directions. The primaryimaging device can be oriented substantially along a direction of motionof the vehicle. The plurality of imaging devices can be located at threeor more of the following locations: a front side, rear side, left side,right side, upper side, or lower side of the vehicle.

In some embodiments, the plurality of images can comprise a plurality ofsuccessive image frames captured over a predetermined time interval. Thepredetermined time interval can be within a range from about 0.02seconds to about 0.1 seconds.

In some embodiments, the image quality of steps (a) and (b) can be eachbased on a feature point number in each image of said plurality ofimages. The feature point number in each image can be calculated using acorner detection algorithm. The corner detection algorithm can be aFeatures from Accelerated Segmented Test (FAST) algorithm.

In some embodiments, the image quality of steps (a) and (b) can be eachbased on saliency of each image of said plurality of images. The imagequality of steps (a) and (b) can be each based on at least one of anexposure level or contrast level in each image of said plurality ofimages. The image quality of steps (a) and (b) can be each based onsuitability of said plurality of images for use in assessing the stateinformation for the vehicle.

In some embodiments, step (b) can comprise assessing whether the imagequality of said plurality of images exceeds a second predeterminedthreshold. Step (b) can comprise identifying which of said plurality ofimages has the highest image quality.

In some embodiments, the state information can comprise at least one ofa position, an attitude, a velocity, or an acceleration of the vehicle.The attitude can comprise at least one of a roll orientation, a pitchorientation, or a yaw orientation of the vehicle.

In some embodiments, steps (a)-(d) can be repeated during operation ofthe vehicle. The steps (a)-(d) can be repeated about once every 0.02seconds to about once every 0.1 seconds.

In another aspect, a method for assessing state information of a movingvehicle attached to a plurality of imaging devices is provided. Themethod comprises: (a) capturing a plurality of images with each imagingdevice of the plurality of imaging devices, and wherein the plurality ofimaging devices comprises a primary imaging device and one or moresecondary imaging devices; (b) assessing, with aid of a processor, imagequality of the plurality of images from the primary imaging device todetermine whether said image quality meets a predetermined threshold;(c) assessing, with aid of the processor, image quality of the pluralityof images from the one or more secondary imaging device if the imagequality of step (b) does not meet the predetermined threshold; (d)selecting, with aid of the processor, at least one of the one or moresecondary imaging devices based on the assessment of step (c); and (e)assessing, with aid of the processor, state information for the vehicleusing the plurality of images from the selected secondary imagingdevice(s).

In some embodiments, the vehicle can be an unmanned aerial vehicle. Theunmanned aerial vehicle may weigh no more than 10 kg. The maximumdimension of the unmanned aerial vehicle may be no more than 1.5 m. Theunmanned aerial vehicle can be configured to fly at a height of no morethan 400 m. Optionally, the unmanned aerial vehicle can be configured todetect the presence of a restricted flight region and not fly within apredetermined distance of the restricted flight region. The restrictedflight region may be an airport. The unmanned aerial vehicle can be amulti-rotor aircraft.

In some embodiments, the one or more propulsion units can comprise oneor more rotors configured to provide lift to the vehicle.

In some embodiments, the plurality of imaging devices are arranged onthe vehicle such that each of the plurality of images is captured from adifferent field of view. In some embodiments, the plurality of imagingdevices can comprise at least three imaging devices. The plurality ofimaging devices can comprise at least four imaging devices. Theplurality of imaging devices can be each oriented in a differentdirection relative to the vehicle. The different directions can beorthogonal directions. Alternatively or in combination, the differentdirections can comprise at least four different directions. The primaryimaging device can be oriented substantially along a direction of motionof the vehicle. The plurality of imaging devices can be located at threeor more of the following locations: a front side, rear side, left side,right side, upper side, or lower side of the vehicle.

In some embodiments, the plurality of images can comprise a plurality ofsuccessive image frames captured over a predetermined time interval. Thepredetermined time interval can be within a range from about 0.02seconds to about 0.1 seconds.

In some embodiments, the image quality of steps (b) and (c) can be eachbased on a feature point number in each image of said plurality ofimages. The feature point number in each image can be calculated using acorner detection algorithm. The corner detection algorithm can be aFeatures from Accelerated Segmented Test (FAST) algorithm.

In some embodiments, the image quality of steps (b) and (c) can be eachbased on saliency of each image of said plurality of images. The imagequality of steps (b) and (c) can be each based on at least one of anexposure level or contrast level in each image of said plurality ofimages. The image quality of steps (b) and (c) can be each based onsuitability of said plurality of images for use in assessing the stateinformation for the vehicle.

In some embodiments, step (c) can comprise assessing whether the imagequality of said plurality of images exceeds a second predeterminedthreshold. Step (c) can comprise identifying which of said plurality ofimages has the highest image quality.

In some embodiments, the state information can comprise at least one ofa position, an attitude, a velocity, or an acceleration of the vehicle.The attitude can comprise at least one of a roll orientation, a pitchorientation, or a yaw orientation of the vehicle.

In some embodiments, steps (a)-(e) can be repeated during operation ofthe vehicle. The steps (a)-(e) can be repeated about once every 0.02seconds to about once every 0.1 seconds.

It shall be understood that different aspects of the invention can beappreciated individually, collectively, or in combination with eachother. Various aspects of the invention described herein may be appliedto any of the particular applications set forth below or for any othertypes of movable objects. Any description herein of an aerial vehiclemay apply to and be used for any movable object, such as any vehicle.Additionally, the systems, devices, and methods disclosed herein in thecontext of aerial motion (e.g., flight) may also be applied in thecontext of other types of motion, such as movement on the ground or onwater, underwater motion, or motion in space. Furthermore, anydescription herein of a rotor or rotor assembly may apply to and be usedfor any propulsion system, device, or mechanism configured to generate apropulsive force by rotation (e.g., propellers, wheels, axles).

In another aspect, a system for navigating a vehicle within anenvironment is provided. The system comprises a vehicle comprising aplurality of sensors arranged on the vehicle such that each sensor ofthe plurality of sensors is configured to obtain sensor data from adifferent field of view; one or more processors operably coupled to theplurality of sensors and individually or collectively configured to: (a)select a subset of the plurality of sensors to be used for navigatingthe vehicle within the environment based on one or more predeterminedcriteria; (b) process the sensor data from the selected sensor(s) so asto generate navigation information for navigating the vehicle within theenvironment; and (c) output signals for controlling the vehicle based onthe navigation information.

In some embodiments, the subset is selected by at least assessing eachsensor to determine if said sensor meets the one or more predeterminedcriteria and selecting said sensor if said sensor meets the one or morepredetermined criteria. The one or more predetermined criteria cancomprise whether said sensor is oriented substantially along a directionof motion of the vehicle. The one or more predetermined criteria cancomprise whether quality of the sensor data obtained by said sensorexceeds a predetermined threshold. The one or more predeterminedcriteria can comprise whether said sensor has the highest sensor dataquality of the plurality of sensors. The one or more predeterminedcriteria can comprise whether power consumption of said sensor is belowa predetermined threshold. The one or more predetermined criteria cancomprise whether said sensor has the lowest power consumption of theplurality of sensors.

In some embodiments, the number of sensors in the subset varies based ona direction of motion of the vehicle. The number of sensors in thesubset can vary based on environmental complexity of the environment.

In some embodiments, the vehicle is an unmanned aerial vehicle. Thevehicle can comprise one or more propulsion units configured to effectmovement of the vehicle. The one or more propulsion units comprise oneor more rotors configured to provide lift to the vehicle.

In some embodiments, the plurality of sensors comprises a plurality ofdifferent sensor types. The plurality of sensors can comprise one ormore of: an imaging device, an ultrasonic sensor, a lidar sensor, or aradar sensor. The plurality of sensors can comprise at least threesensors or at least four sensors. Each sensor can be oriented in adifferent direction relative to the vehicle. The different directionscan be orthogonal directions. The different directions can comprise atleast four different directions. At least one of the differentdirections can be substantially along a direction of motion of thevehicle.

In some embodiments, the plurality of sensors are located at three ormore of the following locations: a front side, rear side, left side,right side, upper side, or lower side of the vehicle.

In some embodiments, the navigation information comprises stateinformation for the vehicle.3 The state information can comprise atleast one of a position, an attitude, a velocity, or an acceleration ofthe vehicle. The attitude can comprise at least one of a rollorientation, a pitch orientation, or a yaw orientation of the vehicle.

In some embodiments, the navigation information comprises environmentalinformation for the environment. The environmental information cancomprise locations of one or more obstacles in the environment. Thesignals can cause the vehicle to avoid the one or more obstacles.

In another aspect, a method for navigating a vehicle within anenvironment is provided. The method comprises: (a) selecting, with aidof a processor, a subset of a plurality of sensors to be used fornavigating the vehicle within the environment based on one or morepredetermined criteria, wherein the plurality of sensors are arranged onthe vehicle such that each sensor of the plurality of sensors isconfigured to obtain sensor data from a different field of view; (b)processing, with aid of the processor, the sensor data from the selectedsensor(s) so as to generate navigation information for navigating thevehicle within the environment; and (c) outputting, with aid of theprocessor, signals for controlling the vehicle based on the navigationinformation.

In some embodiments, selecting the subset comprises assessing eachsensor to determine if said sensor meets the one or more predeterminedcriteria and selecting said sensor if said sensor meets the one or morepredetermined criteria. The one or more predetermined criteria cancomprise whether said sensor is oriented substantially along a directionof motion of the vehicle. The one or more predetermined criteria cancomprise whether quality of the sensor data obtained by said sensorexceeds a predetermined threshold. The one or more predeterminedcriteria can comprise whether said sensor has the highest sensor dataquality of the plurality of sensors. The one or more predeterminedcriteria can comprise whether power consumption of said sensor is belowa predetermined threshold. The one or more predetermined criteria cancomprise whether said sensor has the lowest power consumption of theplurality of sensors.

In some embodiments, the number of sensors in the subset varies based ona direction of motion of the vehicle. The number of sensors in thesubset can vary based on environmental complexity of the environment.

In some embodiments, the vehicle is an unmanned aerial vehicle. Thevehicle can comprise one or more propulsion units configured to effectmovement of the vehicle. The one or more propulsion units comprise oneor more rotors configured to provide lift to the vehicle.

In some embodiments, the plurality of sensors comprises a plurality ofdifferent sensor types. The plurality of sensors can comprise one ormore of: an imaging device, an ultrasonic sensor, a lidar sensor, or aradar sensor. The plurality of sensors can comprise at least threesensors or at least four sensors. Each sensor can be oriented in adifferent direction relative to the vehicle. The different directionscan be orthogonal directions. The different directions can comprise atleast four different directions. At least one of the differentdirections can be substantially along a direction of motion of thevehicle.

In some embodiments, the plurality of sensors are located at three ormore of the following locations: a front side, rear side, left side,right side, upper side, or lower side of the vehicle.

In some embodiments, the navigation information comprises stateinformation for the vehicle. The state information can comprise at leastone of a position, an attitude, a velocity, or an acceleration of thevehicle. The attitude can comprise at least one of a roll orientation, apitch orientation, or a yaw orientation of the vehicle.

In some embodiments, the navigation information comprises environmentalinformation for the environment. The environmental information cancomprise locations of one or more obstacles in the environment. Thesignals can cause the vehicle to avoid the one or more obstacles.

Other objects and features of the present invention will become apparentby a review of the specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates a UAV operating in an outdoor environment, inaccordance with embodiments;

FIG. 2 illustrates a UAV operating in an indoor environment, inaccordance with embodiments;

FIG. 3 illustrates a UAV coupled with multiple sensors, in accordancewith embodiments;

FIG. 4 illustrates sensors with different fields of view, in accordancewith embodiments.

FIG. 5 illustrates a method for navigating a vehicle within anenvironment, in accordance with embodiments;

FIG. 6 illustrates a system for processing state information of a UAVusing images from a plurality of imaging devices, in accordance withembodiments;

FIG. 7 illustrates a method for assessing state information andcontrolling a vehicle based on the processed information, in accordancewith embodiments;

FIG. 8 illustrates a method for assessing state information based oncalculating a feature point number, in accordance with embodiments;

FIG. 9 illustrates a top view of a UAV in motion with a primary imagingdevice and secondary imaging devices, in accordance with embodiments;

FIG. 10 illustrates a method for assessing state information based onprimary and secondary imaging devices, in accordance with embodiments;

FIG. 11 illustrates exemplary feature point numbers for three imagesequences obtained by three secondary imaging devices, in accordancewith embodiments;

FIG. 12 illustrates a UAV, in accordance with embodiments;

FIG. 13 illustrates a movable object including a carrier and a payload,in accordance with embodiments; and

FIG. 14 illustrates a system for controlling a movable object, inaccordance with embodiments.

DETAILED DESCRIPTION

The present disclosure provides navigation systems and methods forassessing state information of vehicles, such as an unmanned aerialvehicle (UAV). In some embodiments, the UAV can be adapted to carry aplurality of sensors configured to collect environmental data. In someembodiments, the sensors may comprise imaging devices such as cameras.The approaches described herein can involve using more than one imagingdevice to obtain a plurality of images with differing fields of view. Insome embodiments, the quality of image frames taken by the plurality ofimaging devices may be compared over a time interval to determine aprimary imaging device with the highest quality image sequence for thatperiod of time. A processor may process information from only the imageframes taken by the primary imaging device to determine or assess thestate for the UAV. This approach may be advantageous for providingaccurate and reliable state information in situations where it isdifficult for a single imaging device to obtain high quality images overa period of time. Processing images for state information from onlyimages captured by the primary imaging device may reduce processing timeand delay, and reduce computer resources that are necessary. The use ofmultiple imaging devices for collecting environmental data as disclosedherein can improve the accuracy of determination of state informationfor the UAV even in diverse environments and operating conditions,thereby enhancing the robustness and flexibility of UAV functionalitiessuch as navigation and obstacle avoidance.

The embodiments provided herein can be applied to various types ofvehicles, such as UAVs. For instance, the UAV may be a small-scale UAVthat weighs no more than 10 kg and/or has a maximum dimension of no morethan 1.5 m. In some embodiments, the UAV may be a rotorcraft, such as amulti-rotor aircraft that is propelled to move through the air by aplurality of propellers (e.g., a quadcopter). Additional examples ofUAVs, other vehicles, and other movable objects suitable for use withthe embodiments presented herein are described in further detail below.

The UAVs described herein can be operated completely autonomously (e.g.,by a suitable computing system such as an onboard controller),semi-autonomously, or manually (e.g., by a human user). The UAV canreceive commands from a suitable entity (e.g., human user or autonomouscontrol system) and respond to such commands by performing one or moreactions. For example, the UAV can be controlled to take off from theground, move within the air (e.g., with up to three degrees of freedomin translation and up to three degrees of freedom in rotation), move totarget location or to a sequence of target locations, hover within theair, land on the ground, and so on. As another example, the UAV can becontrolled to move at a specified velocity and/or acceleration (e.g.,with up to three degrees of freedom in translation and up to threedegrees of freedom in rotation) or along a specified movement path.Furthermore, the commands can be used to control one or more UAVcomponents, such as the components described herein (e.g., sensors,actuators, propulsion units, payload, etc.). For instance, some commandscan be used to control the position, orientation, and/or operation of aUAV payload such as a camera. Optionally, the UAV can be configured tooperate in accordance with one or more predetermined operating rules.The operating rules may be used to control any suitable aspect of theUAV, such as the position (e.g., latitude, longitude, altitude),orientation (e.g., roll, pitch yaw), velocity (e.g., translationaland/or angular), and/or acceleration (e.g., translational and/orangular) of the UAV. For instance, the operating rules can be designedsuch that the UAV is not permitted to fly beyond a threshold height,e.g., the UAV can be configured to fly at a height of no more than 400 mfrom the ground. In some embodiments, the operating rules can be adaptedto provide automated mechanisms for improving UAV safety and preventingsafety incidents. For example, the UAV can be configured to detect arestricted flight region (e.g., an airport) and not fly within apredetermined distance of the restricted flight region, thereby avertingpotential collisions with aircraft and other obstacles.

FIG. 1 illustrates a UAV 102 operating in an outdoor environment 100, inaccordance with embodiments. The outdoor environment 100 may be anurban, suburban, or rural setting, or any other environment that is notat least partially within a building. The UAV 102 may be operatedrelatively close to the ground 104 (e.g., low altitude) or relativelyfar from the ground 104 (e.g., high altitude). For example, a UAV 102operating less than or equal to approximately 10 m from the ground maybe considered to be at low altitude, while a UAV 102 operating atgreater than or equal to approximately 10 m from the ground may beconsidered to be at high altitude.

In some embodiments, the outdoor environment 100 includes one or moreobstacles 108 a-d. An obstacle may include any object or entity that mayobstruct the movement of the UAV 102. Some obstacles may be situated onthe ground 104 (e.g., obstacles 108 a, 108 d), such as buildings, groundvehicles (e.g., cars, motorcycles, trucks, bicycles), human beings,animals, plants (e.g., trees, bushes), and other manmade or naturalstructures. Some obstacles may be in contact with and/or supported bythe ground 104, water, manmade structures, or natural structures.Alternatively, some obstacles may be wholly located in the air 106(e.g., obstacles 108 b, 108 c), including aerial vehicles (e.g.,airplanes, helicopters, hot air balloons, other UAVs) or birds. Aerialobstacles may not be supported by the ground 104, or by water, or by anynatural or manmade structures. An obstacle located on the ground 104 mayinclude portions that extend substantially into the air 106 (e.g., tallstructures such as towers, skyscrapers, lamp posts, radio towers, powerlines, trees, etc.).

FIG. 2 illustrates a UAV 152 operating in an indoor environment 150, inaccordance with embodiments. The indoor environment 150 is within theinterior of a building 154 having a floor 156, one or more walls 158,and/or a ceiling or roof 160. Exemplary buildings include residential,commercial, or industrial buildings such as houses, apartments, offices,manufacturing facilities, storage facilities, and so on. The interior ofthe building 154 may be completely enclosed by the floor 156, walls 158,and ceiling 160 such that the UAV 152 is constrained to the interiorspace. Conversely, at least one of the floor 156, walls 158, or ceiling160 may be absent, thereby enabling the UAV 152 to fly from inside tooutside, or vice-versa. Alternatively or in combination, one or moreapertures 164 may be formed in the floor 156, walls 158, or ceiling 160(e.g., a door, window, skylight).

Similar to the outdoor environment 100, the indoor environment 150 caninclude one or more obstacles 162 a-d. Some obstacles may be situated onthe floor 156 (e.g., obstacle 162 a), such as furniture, appliances,human beings, animals, plants, and other manmade or natural objects.Conversely, some obstacles may be located in the air (e.g., obstacle 162b), such as birds or other UAVs. Some obstacles in the indoorenvironment 150 can be supported by other structures or objects.Obstacles may also be attached to the ceiling 160 (e.g., obstacle 162c), such as light fixtures, ceiling fans, beams, or otherceiling-mounted appliances or structures. In some embodiments, obstaclesmay be attached to the walls 158 (e.g., obstacle 162 d), such as lightfixtures, shelves, cabinets, and other wall-mounted appliances orstructures. Notably, the structural components of the building 154 canalso be considered to be obstacles, including the floor 156, walls 158,and ceiling 160.

The obstacles described herein may be substantially stationary (e.g.,buildings, plants, structures) or substantially mobile (e.g., humanbeings, animals, vehicles, or other objects capable of movement). Someobstacles may include a combination of stationary and mobile components(e.g., a windmill). Mobile obstacles or obstacle components may moveaccording to a predetermined or predictable path or pattern. Forexample, the movement of a car may be relatively predictable (e.g.,according to the shape of the road). Alternatively, some mobileobstacles or obstacle components may move along random or otherwiseunpredictable trajectories. For example, a living being such as ananimal may move in a relatively unpredictable manner.

In order to ensure safe and efficient operation, it may be beneficial toprovide the UAV with mechanisms for assessing environmental informationsuch as the locations of objects in the surrounding environment and/orUAV state information such as position, velocity, attitude, andacceleration. Additionally, accurate assessment of environmental and/orstate information can facilitate navigation, particularly when the UAVis operating in a semi-autonomous or fully autonomous manner and can bevaluable for a wide variety of UAV functionalities.

Accordingly, the UAVs described herein can include one or more sensorsconfigured to collect sensor data that can be processed to obtainnavigation information for navigating the UAV. Navigation informationcan include information relating to the UAV state, the surroundingenvironment, or the objects within the environment. Based on the sensordata that is collected, it can be possible to generate control signalsfor controlling UAV navigation. Exemplary sensors suitable for use withthe embodiments disclosed herein include location sensors (e.g., globalpositioning system (GPS) sensors, mobile device transmitters enablinglocation triangulation), vision sensors (e.g., imaging devices capableof detecting visible, infrared, or ultraviolet light, such as cameras),proximity or range sensors (e.g., ultrasonic sensors, lidar,time-of-flight or depth cameras, radar), inertial sensors (e.g.,accelerometers, gyroscopes, inertial measurement units (IMUs)), altitudesensors, attitude sensors (e.g., compasses) pressure sensors (e.g.,barometers), audio sensors (e.g., microphones) or field sensors (e.g.,magnetometers, electromagnetic sensors). Any suitable number andcombination of sensors can be used, such as one, two, three, four, five,six, seven, eight, or more sensors. Optionally, the data can be receivedfrom sensors of different types (e.g., two, three, four, five, six,seven, eight, or more types). Sensors of different types may measuredifferent types of signals or information (e.g., position, orientation,velocity, acceleration, proximity, pressure, etc.) and/or utilizedifferent types of measurement techniques to obtain data. For instance,the sensors may include any suitable combination of active sensors(e.g., sensors that generate and measure energy from their own energysource) and passive sensors (e.g., sensors that detect availableenergy). As another example, some sensors may generate absolutemeasurement data that is provided in terms of a global coordinate system(e.g., position data provided by a GPS sensor, attitude data provided bya compass or magnetometer), while other sensors may generate relativemeasurement data that is provided in terms of a local coordinate system(e.g., relative angular velocity provided by a gyroscope; relativetranslational acceleration provided by an accelerometer; relativeattitude information provided by a vision sensor; relative distanceinformation provided by an ultrasonic sensor, lidar, or time-of-flightcamera). In some instances, the local coordinate system may be a bodycoordinate system that is defined relative to the UAV.

The sensors can be configured to collect various types of sensor datauseful for navigation, such as data relating to the UAV, the surroundingenvironment, or objects within the environment. The sensor data can beprocessed (e.g., by one or more processors) so as to obtain navigationinformation, such as state information or environmental information. Forexample, at least some of the sensors may be configured to provide dataregarding a state of the UAV and the sensor data obtained by suchsensors can be processed to obtain state information for the UAV. Thestate information provided by a sensor can include information regardinga spatial disposition of the UAV (e.g., location or position informationsuch as longitude, latitude, and/or altitude; orientation or attitudeinformation such as roll, pitch, and/or yaw). The state information canalso include information regarding motion of the UAV (e.g.,translational velocity, translation acceleration, angular velocity,angular acceleration, etc.). A sensor can be configured, for instance,to determine a spatial disposition and/or motion of the UAV with respectto up to six degrees of freedom (e.g., three degrees of freedom inposition and/or translation, three degrees of freedom in orientationand/or rotation). The state information may be provided relative to aglobal coordinate system or relative to a local coordinate system (e.g.,relative to the UAV or another entity). For example, a sensor can beconfigured to determine the distance between the UAV and the usercontrolling the UAV, or the distance between the UAV and the startingpoint of flight for the UAV.

Alternatively or in addition, the data obtained by the sensors may beprocessed to provide various types of environmental information. Forexample, the sensor data may be indicative of an environment type, suchas an indoor environment, outdoor environment, low altitude environment,or high altitude environment. The sensor data may also provideinformation regarding current environmental conditions, includingweather (e.g., clear, rainy, snowing), visibility conditions, windspeed, time of day, and so on. Furthermore, the environmentalinformation collected by the sensors may include information regardingthe objects in the environment, such as the obstacles described herein.Obstacle information may include information regarding the number,density, geometry, and/or location of one or more obstacles in theenvironment.

In some embodiments, the environmental information can includeinformation regarding the complexity of the surrounding environment.“Environmental complexity” may be used herein to refer to the extent towhich an environment is occupied by obstacles. The environmentalcomplexity may be a quantitative or qualitative measure. In someembodiments, the environmental complexity is determined based on one ormore of: the number of obstacles, the volume or percentage of spaceoccupied by obstacles, the volume or percentage of space within acertain proximity to the UAV occupied by obstacles, the volume orpercentage of space unobstructed by obstacles, the volume or percentageof space within a certain proximity to the UAV unobstructed byobstacles, the proximity of obstacles to the UAV, the obstacle density(e.g., number of obstacles per unit space), the types of obstacles(e.g., stationary or mobile), the spatial disposition of obstacles(e.g., position, orientation), the motion of obstacles (e.g., velocity,acceleration), and so on. For instance, an environment having arelatively high obstacle density would have high environmentalcomplexity (e.g., indoor environment, urban environment), whereas anenvironment having a relatively low obstacle density would have lowenvironmental complexity (e.g., high altitude environment). As anotherexample, an environment in which a large percentage of space is occupiedby obstacles would have a higher complexity, whereas an environmenthaving a large percentage of unobstructed space would have a lowercomplexity.

In some embodiments, the navigation information obtained from processingthe sensor data is used to generate signals for controlling variousaspects of UAV navigation, such as movement, obstacle avoidance,environmental mapping, and so on. The signals can be output to variousUAV components, e.g., propulsion units, to effect UAV navigation withinan environment. In some embodiments, such navigation can occurautonomously or semi-autonomously, such that little or no user input isrequired to control the UAV.

In order to optimize the navigation of the UAV within an environment, itmay be beneficial to optimize the accuracy of the navigation informationused as a basis for controlling the UAV. The accuracy of the navigationinformation may depend upon the accuracy, reliability, and quality ofthe sensor data used to generate the navigation information. In someinstances, a single sensor may not be capable of providing satisfactorysensor data for UAV operation at all times, e.g., due to device failure,suboptimal sensor data quality, limitations in sensor field of view,etc. Accordingly, in order to ensure that the obtained sensor data issatisfactory for navigation purposes, it can be beneficial to providemultiple sensors on the UAV for navigation purposes, e.g., to provideredundancy, increase the amount of available data for processing, andcompensate for failure or inaccuracies in any single sensor.

FIG. 3 illustrates a UAV 300 coupled with multiple sensors 308, 310, and312, in accordance with embodiments. The UAV 300 can include a vehiclebody 302. Propulsion units 304, 306 can be coupled to the vehicle body302. The UAV 300 can include one or more sensors coupled to the body302, such as sensors 308, 310, 312. Each sensor can have a differentfield of view 314, 316, 318. In the embodiment of FIG. 3, the sensors308, 310 are coupled to the sides of the body 302, while the sensor 312is connected to the vehicle body 302 by a carrier 320 (e.g., a gimbal).While FIG. 3 shows sensors that are coupled to the sides and connectedto the vehicle body by a carrier, it is to be understood that sensors ofthe present disclosure can be situated on any suitable portion of theUAV, such as above, underneath, on the side(s) of, or within a vehiclebody of the UAV. Some sensors can be mechanically coupled to the UAVsuch that the spatial disposition and/or motion of the UAV correspond tothe spatial disposition and/or motion of the sensor. The sensors can becoupled to the UAV via a rigid coupling, such that the sensor does notmove relative to the portion of the UAV to which it is attached.Alternatively, the coupling between the sensor and the UAV can permitmovement (e.g., translational or rotational movement relative to theUAV) of the sensor relative to the UAV. The coupling can be a permanentcoupling or non-permanent (e.g., releasable) coupling. Suitable couplingmethods can include adhesives, bonding, welding, and/or fasteners (e.g.,screws, nails, pins, etc.). Optionally, the sensor can be integrallyformed with a portion of the UAV. Furthermore, the sensor can beelectrically coupled with a portion of the UAV (e.g., processing unit,control system, data storage) so as to enable the data collected by thesensor to be used for various functions of the UAV (e.g., navigation,control, propulsion, communication with a user or other device, etc.),such as the embodiments discussed herein. The sensor may be operablycoupled with a portion of the UAV (e.g., processing unit, controlsystem, data storage).

Although FIG. 3 depicts a UAV with three sensors, it shall be understoodthat the embodiments described herein can be applied to any suitablenumber of sensors, such as one, two, three, four, five, six, seven,eight or more sensors. In some embodiments, the UAV includes at leastone, at least two, at least three, at least four, at least five, atleast six, at least seven, or at least eight sensors. The sensors mayeach be pointing in different directions. The sensors can be positionedand oriented relative to each other and to the UAV as desired. In someembodiments, sensors may be positioned near each other. Alternatively,some may be positioned away from each other. The sensors may bepositioned on opposite sides of the vehicle, on adjacent sides of thevehicle, on the same side of the vehicle, on same portion of thevehicle, or on different portions of the vehicle. For example, eachsensor can be mounted on a different side or surface of the UAV (e.g.,front, rear, left, right, top (upper side), and/or bottom (lower side)surfaces). In some embodiments, some or all of the sensors are mountedon one or more carriers attached to the UAV. In some embodiments, theplurality of sensors can be located at three or more different sides orsurfaces of the UAV (e.g., front, rear, left, right, upper, lower sidesof the vehicle).

The sensors can be situated at different positions and orientations suchthat the field of view of each sensor is different. The field of view ofa sensor may be the extent of the environment that is detectable (e.g.,visible) by the sensor. Sensors with different fields of view may depictdifferent portions of the surrounding environment. The fields of view ofsome or all of the sensors may overlap. Alternatively, the fields ofview of the sensors may not overlap. The fields of view between twosensors may overlap but be different (e.g., only partially overlapping).The field of view may be related to the angle of view, which may bemeasured by the angular extent of a given scene that is imaged by thesensor. The angle of view of an sensor may be at an angle of less thanor about 360°, 300°, 270°, 240°, 180°, 120°, 90°, 60°, 45°, 30°, 20°,10°, 5°, or 1°. The angle of view of each sensor may be different.Alternatively, the angle of view of some or all of the sensors may bethe same.

In some embodiments, the directionality of a sensor with opticalcomponents (e.g., vision sensors such as cameras) can be characterizedby the directionality of its optical axis. A plurality of sensors can besituated at different positions and orientations such that theirrespective optical axes are different. The optical axis of a sensor,which may also be referred to as the “principal axis,” can be a linealong which there is some degree of rotational symmetry in the sensor.In some embodiments, the optical axis of the sensor passes through thecenter of the optical components (e.g., lens, photo sensor) of thesensor. In some embodiments, the sensors may be arranged such that theirrespective optical axes are at an angle of about 10°, 20°, 30°, 40°,50°, 60°, 70°, 80°, 90°, 100°, 110°, 120°, 130°, 140°, 150°, 160°, 170°,or 180° relative to one another. In some embodiments, the sensors may bespaced evenly apart (e.g., two devices are 180° apart, three devices are120° apart, four devices are 90° apart, etc). In some embodiments, thesensors may be orthogonally oriented with respect to one another.Alternatively or in combination, at least one of the sensors may beoriented along the direction of motion of the UAV. The optical axis maybe the axis from which the angle of view is measured. The angle of viewmay be measured vertically, horizontally, or diagonally along theoptical axis.

FIG. 4 illustrates two sensors 405, 410 positioned near with differentfields of view 415, 420, in accordance with embodiments. In someembodiments, the sensors 405, 410 are cameras. Sensor 405 is obtainingsensor data with a field of view 415 and sensor 410 is obtaining sensordata with a field of view 420. In the embodiment of FIG. 4, the fieldsof view 415, 420 are different and do not overlap. In some embodiments,the angles of view 435, 440 of the sensors 405, 410 measuredhorizontally are about 60°. The sensor 405 can have an optical axis 425that is at a 90° angle relative to the optical axis 430 of sensor 410.In alternative embodiments, the fields of view 415, 420, angles of view435, 440, and/or optical axes 425, 430 can be varied as desired.

The sensors carried on a UAV (or other movable object) may all be of thesame type. Alternatively, at least some of the sensors may be ofdifferent types. In some embodiments, the sensors each provide sensordata of the same scene from different positions and/or orientations. Thesensors can be configured to capture sensor data of a scene at the sametime or approximately the same time. Alternatively, some sensors can beconfigured to capture sensor data at different times than other sensors.

As previously described herein, the use of multiple sensors forcollecting sensor data can improve the accuracy of navigationinformation in some instances. However, in certain situations,processing sensor data from multiple sensors may require more processingpower, processing time, and/or computing resources than is ideal. Forexample, small-scale UAVs (e.g., weighing less than 10 kg) and othersmall movable objects may not be capable of carrying sufficientprocessors, memory, and/or other types of computing resources to enableprocessing of multiple sensor data sets at a sufficient speed fornavigation purposes.

Accordingly, in some embodiments, selective processing of sensor data isused to conserve processing power and computing resources and reduceprocessing time. In such embodiments, only a subset of the sensorscarried by the UAV are used to provide sensor data for generatingnavigation information. The subset of sensors can be selected based onone or more predetermined criteria so as to optimize the accuracy of theresultant navigation information, as well as to improve overall UAVperformance (e.g., reduce power consumption, processor load, calculationcomplexity). This approach can improve the speed and efficiency ofsensor data processing while maintaining improved accuracy andreliability of navigation information.

FIG. 5 illustrates a method 500 for navigating a vehicle within anenvironment, in accordance with embodiments. The method 500 can beapplied to control a vehicle such as a UAV carrying a plurality ofsensors (e.g., imaging devices, ultrasonic sensors, lidar sensors, radarsensors, or combinations thereof). Optionally, the sensors can bearranged on the vehicle such that each sensor obtains respective sensordata from a different field of view. At least some of the fields of viewmay overlap with each other. In some embodiments, at least some of thefields of view are different. In other embodiments, at least some of thefields of view are the same. Some or all of the steps of the method 500can be performed by one or more processors that are operably coupled tothe plurality of sensors. In some embodiments, the method 500 isperformed without requiring user input, thereby enabling autonomousselection and processing of sensor data for generating navigationinformation.

In step 510, a subset of a plurality of sensors carried by the vehicleis selected, based on one or more predetermined criteria. The subset caninclude any number of sensors, such as at least one, two, three, four,five, or more sensors. In some embodiments, the subset is less than allof the sensors carried by the vehicle for navigation. In someembodiments, the subset includes only a single sensor. The number ofsensors to be selected for the subset can vary based on certain factors.For example, the number of sensors in the subset can vary based on thecharacteristics of the surrounding environment, such as theenvironmental complexity. A larger number of sensors can be selectedwhen the vehicle is operating within an environment that is relativelycomplex (e.g., high obstacle density), whereas a smaller number ofsensors can be selected when the vehicle is operating within anenvironment that is not complex (e.g., low obstacle density). It may bebeneficial to select a larger number of sensors when navigating withinhighly complex environments in order to provide additional redundancyand accuracy of the sensor data and reduce the risk of accidents (e.g.,collisions with obstacles). In some embodiments, at least 2, 3, 4, 5, ormore sensors are selected when the vehicle is operating within a highlycomplex environment.

As another example, the number of sensors selected as part of the subsetcan vary based on the state of the vehicle, such as the direction and/orthe speed of motion of the vehicle. In some embodiments, a single sensoris selected if the direction of motion is one-dimensional (e.g.,relative to orthogonal coordinate axes of the vehicle reference frame),two sensors are selected if the direction of motion is two-dimensional,and three sensors are selected if the direction of motion isthree-dimensional. For example, a single sensor can be selected when thesensor is aligned with the direction of motion, two sensors can beselected when the sensors are oblique to the direction of motion butstill within the same plane of motion, and three sensors can be selectedwhen the sensors are oblique to the direction of motion and outside theplane of motion. The alignment of the sensors relative to the directionof motion can be determined according to the field of view, angle ofview, and/or optical axis of the sensors, as previously describedherein.

Various types of predetermined criteria can be used to select sensors.For example, the criteria can be related to the orientation and/or fieldof view of the sensor, e.g., whether the sensor is orientedsubstantially along the direction of motion of the vehicle, whether thefield of view of the sensor overlaps the direction of motion, etc. Asanother example, the criteria can be related to the quality of thesensor data produced by the sensor (e.g., signal to noise ratio,suitability for processing, accuracy, robustness, reliability), such aswhether the quality exceeds a predetermined threshold, whether thesensor has the highest sensor data quality, etc. In some embodiments,the sensor data from each sensor is analyzed to determine the qualityand only the sensors producing the highest quality data are selected. Inyet another example, the criteria can be related to the powerconsumption of each sensor, e.g., whether the power consumption is belowa predetermined threshold, whether the sensor has the lowest powerconsumption, etc. In some embodiments, sensors exhibiting lower powerconsumption are preferentially selected before sensors exhibiting higherpower consumption. The one or more predetermined criteria can beprovided in various ways, such as preset or preprogrammed in the UAVprior to operation, transmitted to the UAV during operation, determinedbased on user input before or during operation, or combinations thereof.In some embodiments, the step 510 involves assessing each sensor todetermine if it meets the one or more predetermined criteria, andselecting the sensor if it does meet the criteria.

In step 520, sensor data from the selected sensor(s) is processed inorder to generate navigation information for navigating the vehiclewithin the environment. As previously described herein, the sensor datacan be processed to obtain state information for the vehicle and/orenvironmental information for the environment. In some embodiments, thenavigation information is determined using only the sensor data from theselected sensor(s), such that sensor data from the remaining sensors isnot processed. This selective processing approach can improve theprocessing efficiency and speed for determining navigation information,as previously described herein.

In step 530, signals for controlling the vehicle are output based on thenavigation information generated in step 530. The signals can beconfigured for controlling navigation of the vehicle within theenvironment. In some embodiments, the signals include control signalsfor the propulsion units of the vehicle in order to control the spatialdisposition and/or movement of the vehicle. For example, environmentalinformation regarding the location of one or more obstacles can be usedto generate control signals to cause the vehicle to move in a mannerthat avoids the obstacles. In another example, state informationregarding a current velocity of the vehicle can be used as feedback toadjust the operation of the propulsion units in order to ensure that thevehicle achieves and maintains a desired velocity.

In some embodiments, the approaches described herein can be applied toselective processing of image data obtained by vision sensors, alsoreferred to herein as “imaging devices.” An imaging device can beconfigured to detect electromagnetic radiation (e.g., visible, infrared,and/or ultraviolet light) and generate image data based on the detectedelectromagnetic radiation. For example, an imaging device may include acharge-coupled device (CCD) sensor or a complementarymetal-oxide-semiconductor (CMOS) sensor that generates electricalsignals in response to wavelengths of light. The resultant electricalsignals can be processed to produce image data. The image data generatedby an imaging device can include one or more images, which may be staticimages (e.g., photographs), dynamic images (e.g., video), or suitablecombinations thereof. The image data can be polychromatic (e.g., RGB,CMYK, HSV) or monochromatic (e.g., grayscale, black-and-white, sepia).Although certain embodiments provided herein are described in thecontext of imaging devices, it shall be understood that the presentdisclosure can be applied to any suitable type of sensor, andvice-versa.

In some embodiments, the imaging device can be a camera. A camera can bea movie or video camera that captures dynamic image data (e.g., video).A camera can be a still camera that captures static images (e.g.,photographs). Although certain embodiments provided herein are describedin the context of cameras, it shall be understood that the presentdisclosure can be applied to any suitable imaging device, and anydescription herein relating to cameras can also be applied to anysuitable imaging device, and any description herein relating to camerascan also be applied to other types of imaging devices. A camera can beused to generate 2D images of a 3D scene (e.g., an environment, one ormore objects, etc.). The images generated by the camera can representthe projection of the 3D scene onto a 2D image plane. Accordingly, eachpoint in the 2D image corresponds to a 3D spatial coordinate in thescene.

The images obtained by the imaging devices described herein can be usedfor a variety of applications related to UAV operation. In someembodiments, the images are used to facilitate UAV navigation within anenvironment (e.g., autonomously, semi-autonomously, or manually). Forexample, the images can be processed to determine state information forthe UAV (e.g., position, orientation, velocity, and/or accelerationinformation). Alternatively or in addition, the images can be processedto determine environmental information (e.g., complexity, location ofobstacles). Any description herein referring to state information canalso be applied to other types of navigation information such asenvironmental information, and vice-versa. The state information can bedetermined using images obtained by a single imaging device.Alternatively, the state information can be determined using imagesobtained from multiple imaging devices.

In some embodiments, the imaging devices described herein are configuredto capture a plurality of images that are processed in order todetermine the state information. A plurality of images taken by animaging device may be referred to herein as an image sequence, orsequence of images. An individual image captured by the imaging devicemay be referred to herein as an “image frame.” An image sequence caninclude one or more image frames. The sequence of images can be capturedat a specific capture rate. In some embodiments, the image frames may becaptured standard video frame rates such as about 24p, 25p, 30p, 48p,50p, 60p, 72p, 90p, 100p, 120p, 300p, 50i, or 60i. In some embodiments,the image frames may be captured at a rate less than or equal to aboutonce every 0.0001 seconds, 0.0002 seconds, 0.0005 seconds, 0.001seconds, 0.002 seconds, 0.005 seconds, 0.002 seconds, 0.05 seconds, 0.01seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds. In someembodiments, the capture rate may change depending on user input, stateinformation, or external conditions (e.g. rain, snow, wind, unobvioussurface texture of environment).

The sequence of images may be captured over a time interval. In someembodiments, the image sequence, or plurality of images, may comprise aplurality of successive image frames captured over a predetermined timeinterval. The time interval can be set as desired by a user. The timeinterval may be fixed. Alternatively, the time interval may beautomatically determined, e.g., by a processor. The time interval mayremain fixed or may change during the course of operation of the UAV orits components (e.g., imaging devices). In some embodiments, the timeinterval may be less than or equal to about 0.005 seconds, 0.002seconds, 0.05 seconds, 0.01 seconds, 0.02 seconds, 0.05 seconds, 0.1seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10seconds, 20 seconds, 50 seconds, 100 seconds, 200 seconds, 500 seconds,1000 seconds, or 3600 seconds. In some embodiments, the time intervalmay be within a range from about 0.02 seconds to about 0.1 seconds. Insome embodiments, the time interval may change depending on user input,state information, or external conditions. The imaging devices maycapture said sequence of images for any duration. In some embodiments,duration can be the time during which the UAV is operational, the timeduring which the UAV is moving, time as set by the user, time asautomatically determined by a processor, time as preset, the time duringwhich user input is given, the time during which there is instability(e.g. turbulence, drift), or depending on state information.

In some embodiments, the image sequences captured by the imaging devicescan be processed to detect one or more feature points present in eachimage of the plurality of images from each imaging device. A featurepoint can be a portion of an image (e.g., an edge, corner, interestpoint, blob, ridge, etc.) that is uniquely distinguishable from theremaining portions of the image and/or other feature points in theimage. Optionally, a feature point may be relatively invariant totransformations of the imaged object (e.g., translation, rotation,scaling) and/or changes in the characteristics of the image (e.g.,brightness, exposure). A feature point may be detected in portions of animage that is rich in terms of informational content (e.g., significant2D texture). A feature point may be detected in portions of an imagethat are stable under perturbations (e.g., when varying illumination andbrightness of an image). Feature detection as described herein can beaccomplished using various algorithms which may extract one or morefeature points from image data and calculate the total number of featurepoints, or “feature point number.” The algorithm may be an edgedetection algorithm, a corner detection algorithm, a blob detectionalgorithm, or a ridge detection algorithm. In some embodiments, thecorner detection algorithm may be a “Features from accelerated segmenttest” (FAST). In some embodiments, the feature detector may extractfeature points and calculate a feature point number using FAST. In someembodiments, the feature detector can be a Canny edge detector, Sobeloperator, Harris & Stephens/Plessy/Shi-Tomasi corner detectionalgorithm, the SUSAN corner detector, Level curve curvature approach,Laplacian of Gaussian, Difference of Gaussians, Determinant of Hessian,MSER, PCBR, or Grey-level blobs, ORB, FREAK, or suitable combinationsthereof.

State information can be assessed according to the extracted featuresusing one or more processors. In some embodiments, a processor maycomprise a field-programmable gate array (FPGA), application-specificintegrated circuit (ASIC), application-specific standard product (ASSP),or complex programmable logic devices (CPLD). In some embodiments, theprocessor may be an embedded processor carried by the UAV.Alternatively, the processor may be separated from the UAV (e.g., at aground station, communicating with the UAV). In some embodiments, theprocessor can take the extracted feature points and compare them acrossa sequence of image frames in order to track and map objects anddetermine the relative changes over the image frames. In someembodiments, the processor can determine state information of the UAVsuch as position (e.g. longitude, latitude, and altitude), attitude,orientation (e.g., roll, pitch, and yaw), velocity (e.g. translationaland angular velocity), and acceleration (e.g. translational and angularacceleration) by analyzing the relative changes over the image framesbased on the feature points.

FIG. 6 illustrates a system 600 for processing state information of aUAV using images from a plurality of imaging devices 602, in accordancewith embodiments. The system 600 can include a plurality of imagingdevices 602, a first processor 604, and a second processor 606. Each ofthe imaging devices 602 can be used to capture an image sequence. Theimage sequences captured by the imaging devices 602 can be sent to thefirst processor 604 (e.g., an FPGA) which processes the image sequencesin order to detect feature points in each of the images, e.g., using thealgorithms described above. In some embodiments, the processor 604determines a plurality of feature points from each of the images in theimage sequences, calculates a feature point number, and uses the featurepoint number to assess or determine image quality, as described infurther detail herein. The feature points for each of the imagesequences can be transmitted to the second processor 606 (e.g., anembedded processor). The processor 606 can process the image sequencesfurther to assess the state information of the UAV. Optionally, theprocessor 606 can determine the state information using the featurepoints of the image sequences generated by the processor 604, aspreviously described herein. The state information can be output to aflight control module of the UAV and used as a basis for outputtingcontrol signals to effect movement of the UAV. For example, the controlsignals can be output to the one or more propulsion units mounted on theUAV. Although the system 600 is depicted as including two separateprocessors 604, 606 for detecting feature points and assessing stateinformation, any suitable number and combination of processors can beused. For example, a single processor can be used to perform featurepoint detection and state information determination. In someembodiments, at least some of the processors may be carried by the UAV.In some embodiments, at least some of the processors may be separatefrom the UAV (e.g., at a ground station communicating with the UAV).There may be one or more processors for detecting the feature points andassessing the state information. The feature points and stateinformation can be determined by different processors, or by the sameprocessors.

The accuracy and robustness of the state information determined fromimages may depend on the image quality of the images. In someembodiments, image quality refers to the suitability of the imageframe(s) for image processing (e.g., to assess UAV state information).Poor quality images not suitable for processing may provide inaccuratedata while high quality images suitable for processing may provideaccurate data. Image quality may be related to or dependent on the imagesaliency. Image saliency can be used herein to refer to the extent towhich images have features that are easily distinguishable or “standout,” e.g., from the background and/or surrounding image pixels. Anydescription herein referring to image quality may also be applied toimage saliency, and vice-versa. In some embodiments, image quality canbe determined by the exposure level and/or contrast of the image. Insome embodiments, image quality may be determined using image gradientmethods in which a gradient for each pixel in the image can becalculated and the results can be used to determine whether the imagetexture is sufficiently rich. An image with a richer texture may havelarger gradients which may signify a higher quality image. In someembodiments, image quality is assessed by feature detection, aspreviously described herein. For instance, the image quality can berelated to the number of feature points detected in the image. In someembodiments, the number of features points in an image is assessed andidentified in order to determine the image quality. In some embodiments,the feature point number in each image can be indicative of saliency ofthe image. A high number of feature points can signify a high qualityimage suited to be used in assessing state information while a lownumber of feature points can signify a low quality image not suited tobe used in assessing state information. For example, a high qualityimage may have at least about 100, 150, 200, 250, 500, 1000 or morefeature points. Conversely, a low quality image may have less than about100, 75, 60, 50, 25, or fewer feature points.

In some embodiments, some or all of the image frames captured by asingle imaging device may be of relatively poor image quality, e.g., dueto suboptimal imaging conditions. Imaging devices may not be able toprovide satisfactory image data in certain situations, e.g., when thelighting is bright or has high contrast, or in adverse environmentalconditions such as rain, fog, or smog. Poor image quality can resultwhen images depict repetitive environments (e.g., walls, water),environments with low contrast (e.g., snow, nighttime), overexposure,underexposure, and/or unobvious surface texture. Moreover, even if theinitial sequence of images taken by an imaging device were of goodquality, subsequent images taken by the imaging device may degrade dueto changing environmental conditions. Poor image quality can result in areduced number of feature points, resulting in processing of inaccuratestate information or in some scenarios, the inability to process anystate information.

In some embodiments, poor image quality is caused by overexposure of theimage. An image may be described as overexposed when it has a loss ofhighlight detail and/or when important parts of an image are “washedout.” A processor may only be able to detect feature points from theportion of the overexposed image that was not washed out, leading to theidentification of fewer feature points which may be difficult todistinguish from image noise.

In some embodiments, poor image quality occurs when the image depicts ascene with low or unobvious surface texture (e.g., a blank wall).Surface texture can be the nature of a surface as defined by lay,surface roughness, and waviness. An image with unobvious surface texturemay have relatively low image contrast. An image of a surface withunobvious surface texture may result in fewer feature points that may bedifficult to distinguish from image noise.

To mitigate this problem, a processor may determine state informationusing a plurality of image sequences taken by a plurality of imagingdevices. If feature points are being extracted from many differentfields of view, there may be a greater chance that at least one of theimaging devices produces a high quality image sequence suitable fordetermining the state information. However, using multiple sequences ofimage frames from multiple imaging devices in order to assess stateinformation of the UAV can increase the use of computing resourcesand/or increase processing time. Accordingly, in some embodiments, onlya subset of the multiple imaging devices is used to determine the stateinformation. Various methods can be used to select which imagingdevice(s) to use. For example, the selection can be performed (e.g.,with the aid of a processor) based on the quality of the image sequencesproduced by the imaging devices. The image quality can be determinedbased on the number of feature points present in the images of eachimage sequence, as previously described herein.

In some embodiments, the assessment of image quality for each imagesequence is performed in parallel (e.g., simultaneously), such that theprocessing delay of the system is not necessarily increased even whenprocessing images captured from multiple imaging devices. In someembodiments, the determination of state information from each imagesequence is performed in serial (e.g., consecutively), such that theprocessing delay of the system can be increased if information frommultiple imaging devices is processed to determine state information.Accordingly, it may be beneficial to select only a subset of the imagingdevices to be used for determining state information while imagesequences from all imaging devices are processed to determine imagequality. In some embodiments, only one imaging device is selected.Alternatively, two, three, four, five, six, seven, eight, or moreimaging devices can be selected.

The selection of the imaging devices for determining state informationmay be based on one or more criteria. The criteria can be predeterminedcriteria (e.g., preset prior to operation of the UAV by a user).Alternatively, the criteria can be determined during operation of theUAV, e.g., automatically or based on user input. Optionally, thecriteria can be dynamic such that the criteria can be modified duringoperation of the UAV, e.g., in response to changing operationalconditions. In some embodiments, the criteria may be related to theimage quality or image saliency of the image sequence and/or individualimages in the image sequence. The image quality or image saliency may berelated to the feature point number of the image sequence, as previouslydescribed herein. In some embodiments, the “number of feature points”will be interchangeable with “image quality” for purposes of determiningthe criteria for the selection of imaging devices whose image sequencewill be assessed to determine state information. For example, thecriteria may be whether the overall image sequence or each image in thesequence has met a minimum threshold for image quality (e.g., has met aminimum feature point number). For example, the minimum feature pointnumber used as an image quality threshold may be about 10, 20, 30, 40,50, 60, 70, 80, 90, 100, 150, 200, 500 or more feature points. Thecriteria may be to select an imaging device that produces an imagesequence with the highest average image quality (e.g., highest featurepoint number per image frame). The criteria may be to select an imagingdevice with a highest maximum image quality within the image sequence(e.g., highest maximum feature point number). The criteria may be toselect an imaging device with the highest minimum image quality withinthe image sequence (e.g., the highest minimum feature point number). Itis to be understood that the criteria can change according to the numberof imaging devices that are to be selected. For example, if two imagedevices are to be selected, the criteria may be to select the twoimaging devices with the first and second highest maximum image qualitywithin the image sequence (e.g., first and second highest feature pointnumbers).

Referring again to FIG. 6, in some embodiments, one or more processorsof the system 600 can assess the image quality of the captured imageframes. For example, the processor 604 can be used to process the imagesequences from each of the imaging devices 602 in order to assess theimage quality of each image sequence. The processor 604 can process eachimage frame that is sent to it in order to determine an image quality.For example, in some embodiments, the processor 604 determines aplurality of feature points from each of the images in the imagesequences, calculates a feature point number, and uses the feature pointnumber to determine image quality. The feature point number for eachsequence can be transmitted to the processor 606, which can use thefeature point numbers as a basis for selecting the imaging device(s) forassessing state information. The image sequences obtained by selectedimaging device(s) can then be processed by the processor 606 to generatestate information.

FIG. 7 illustrates a method 1000 for assessing state information andcontrolling a vehicle based on the processed information, in accordancewith embodiments. The steps of the method 1000, as with all methodspresented herein, can be performed using any embodiment of the systemsand devices described herein. For example, the method 1000 can beperformed by a navigation system carried by the UAV. In someembodiments, steps 1010 through 1050 of the method 1000 can be performedby one or more processors, at least some of which may be carried by theUAV. Furthermore, at least some of the steps may be performedautomatically without requiring user input. Alternatively or incombination, at least some of the steps may involve user input. Some orall of the steps of the method 1000 can be repeated as desired. Forexample, the steps 1010 through 1050 may be repeated at a desired rate.The rate may be less than or equal to about once every 0.005 seconds,0.01 seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 50seconds, 100 seconds, 200 seconds, 500 seconds, 1000 seconds, or 3600seconds. In some embodiments, the rate may be about once every 0.02seconds to about once every 0.1 seconds.

In step 1010, a plurality of sequence of images can be captured by theplurality of imaging devices over a time interval. The time interval maybe less than or equal to about 0.005 seconds, 0.01 seconds, 0.02seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second,2 seconds, 5 seconds, 10 seconds, 20 seconds, 50 seconds, 100 seconds,200 seconds, 500 seconds, 1000 seconds, or 3600 seconds. In someembodiments, the time interval may be within a range from about 0.02seconds to about 0.1 seconds. In some embodiments, the time interval maybe within a range from about 0.005 seconds to about 10 seconds. An imagesequence captured during the time interval can be comprised of about 1image, 2 images, 5 images, 10 images, 20 images, 50 images, 100 images,200 images, 500 images, or 1000 images. Optionally, the imaging devicesmay be arranged on the UAV such that each of the plurality of images iscaptured from a different field of view, as previously discussed herein.

In step 1020, the image quality of the plurality of images from eachimaging device can be assessed with the aid of a processor. The imagequality can be related to the feature point number, as previouslydescribed herein. The determination of image quality can be processed inparallel (e.g., all image sequences simultaneously). Thus the processingdelay of the system may not be necessarily increased even if imagescaptured from multiple imaging devices are processed for image quality.

In step 1030, at least one of the image sequences taken by the pluralityof imaging devices can be selected based on the assessment of imagequality of step 1020. The selection of the imaging devices may be basedon one or more criteria. The criteria may be related to the imagequality or image saliency of the image sequences, as previouslydescribed herein. For example, the criteria may be whether the overallimage sequence or each image in the sequence met a minimum threshold forimage quality. The criteria may be to select an imaging device that hadan image sequence with the highest average image quality. The criteriamay be to select an imaging device with a highest maximum image qualitywithin the image sequence. The criteria may be to select an imagingdevice with the highest minimum image quality within the image sequence.It is to be understood that the criteria may change according to thenumber of imaging devices that are to be selected. For example, ifselecting two image devices, the criteria may be to select the twoimaging devices with the first and second highest maximum image qualitywithin the image sequence. In some embodiments, the “number of featurepoints” will be interchangeable with “image quality” for purposes ofdetermining the criteria for the selection of imaging devices whoseimage sequence will be assessed to determine state information.

In step 1040, the state information for the UAV can be assessed with theaid of the processor, using the plurality of images from the imagingdevice(s) selected in step 1030. An assessment or determination of stateinformation from the processor can be performed in serial (e.g., oneimage sequence at a time, consecutively). Thus the processing delay ofthe system can be increased if information from multiple imaging devicesis processed. By having a criterion for selecting an image sequence fromonly a subset of the imaging devices to be processed in determiningstate information, processing time and computing resources can be saved.In some embodiments, while image quality is assessed for all imagestaken by the plurality of imaging devices as it is processed inparallel, the state information is assessed using data from only asubset of the imaging devices because it is serially processed.Moreover, the determination of state information can be morecomputationally intensive than image quality determination which can bea low level image processing operation that is computationallyinexpensive. Thus, FIG. 7 can illustrate a method of determining stateinformation that is computationally and temporally efficient.

In step 1050, the state information can optionally be used as a basisfor outputting signals to cause the UAV to navigate within theenvironment. The signal can include control signals for the propulsionsystem (e.g., rotors) of the UAV for effecting movement of the vehicle.The signal can be generated based on user commands that are input into aremote terminal or other user device and subsequently transmitted to theUAV. Alternatively, the signal can be autonomously generated by the UAV(e.g., an automated onboard controller). In some instances, the signalcan be generated semi-autonomously with contributions from user input aswell as being automated. In some embodiments, the state information isused as input for a flight control algorithm that produces the controlsignals.

FIG. 8 illustrates a method 1100 for assessing state information basedon calculating a feature point number, in accordance with embodiments.The method 1100 can be understood as a specialized version of the method1000 in which the image quality of the plurality of images taken by eachimaging device is assessed by calculating a feature point number in eachimage of the images.

The steps 1110 through 1150 of the method 1100 can be performed by oneor more processors, at least some of which may be carried by the UAV. Insome embodiments, the method 1100 is performed by a navigation system ofthe UAV. Furthermore, at least some of the steps may be performedautomatically without requiring user input. Alternatively or incombination, at least some of the steps may involve user input. Some orall of the steps of the method 1100 can be repeated as desired. Forexample, the steps 1110 through 1150 may be repeated at a desired rate.The rate may be less than or equal to about once every 0.005 seconds,0.01 seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 50seconds, 100 seconds, 200 seconds, 500 seconds, 1000 seconds, or 3600seconds. In some embodiments, the rate may be about once every 0.02seconds to about once every 0.1 seconds.

In step 1110, a plurality of sequence of images can be captured by eachimaging device of the plurality of imaging devices over a time interval.The time interval may be less than or equal to about 0.005 seconds, 0.01seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 50seconds, 100 seconds, 200 seconds, 500 seconds, 1000 seconds, or 3600seconds. In some embodiments, the time interval may be within a rangefrom about 0.02 seconds to about 0.1 seconds. In some embodiments, thetime interval may be within a range from about 0.005 seconds to about 10seconds. An image sequence captured during the time interval can becomprised of about 1 image, 2 images, 5 images, 10 images, 20 images, 50images, 100 images, 200 images, 500 images, or 1000 images. The imagingdevices may be arranged on the UAV such that each of the plurality ofimages is captured from a different field of view, as previouslydiscussed herein.

In step 1120, a feature point number is calculated in each image of theplurality of images from each imaging device, e.g., with the aid of aprocessor. In some embodiments, the feature point number may be assessedby calculating a feature point number in each image, using a featuredetection algorithm such as FAST. The calculation of the feature pointnumber can be processed in parallel. Thus the processing delay of thesystem may not be necessarily increased even if images captured frommultiple imaging devices are processed for a feature point number.

In step 1130, at least one of the plurality of imaging devices can beselected based on the assessment of step 1120. The selection of theimaging devices may be based on one or more criteria. The criteria maybe related to the feature point number of the image sequences. Forexample, the criteria may be whether the overall image sequence or eachimage in the sequence met a minimum threshold for a feature pointnumber. The criteria may be to select an imaging device that had animage sequence with the highest average feature point number. Thecriteria may be to select an imaging device with a highest maximumfeature point number within the image sequence. The criteria may be toselect an imaging device with the highest minimum feature point numberwithin the image sequence. It is to be understood that the criteria maychange according to the number of imaging devices that are to beselected. For example, if selecting two image devices, the criteria maybe to select the two imaging devices with the first and second highestmaximum feature point number within the image sequence.

In step 1140, the state information for the UAV can be assessed with theaid of the processor, using the plurality of images selected in step1130. An determination of state information from the processor can beperformed in serial. Thus the processing delay of the system can beincreased if information from multiple imaging devices is processed. Byhaving a criterion for selecting a subset of the images to be processedin determining state information, processing time and computingresources can be saved.

Optionally, the state information can be used as a basis for outputtingsignals to cause the UAV to navigate within the environment. The signalcan include control signals for the propulsion system (e.g., rotors) ofthe UAV for effecting movement of the vehicle. The signal can begenerated based on user commands that are input into a remote terminalor other user device and subsequently transmitted to the UAV.Alternatively, the signal can be autonomously generated by the UAV(e.g., an automated onboard controller implementing suitable flightcontrol algorithms). In some instances, the signal can be generatedsemi-autonomously with contributions from user input as well as beingautomated.

In some embodiments, one or more imaging devices are designated as the“default” imaging devices to be used for assessing state information.Such imaging devices may be referred to herein as “primary imagingdevices.” Imaging devices that are not selected by default may bereferred to herein as “secondary imaging devices.” A primary or defaultimaging device can be an imaging device whose image sequence is alwaysused in order to determine image quality and whose image sequence isused to assess state information of the UAV if the image quality isdeemed satisfactory. A secondary or non-default imaging device can be animaging device whose image sequence is assessed for image quality onlyif the image quality of the image sequence taken by the primary imagingdevice is deemed unsatisfactory. An image sequence taken by a secondaryimaging device may be used to assess state information of the UAV onlyif an image sequence taken by the primary imaging device is ofunsatisfactory image quality. A UAV can have any suitable number ofprimary and secondary imaging devices. For example, a UAV may have one,two, three, four, five, six, seven, eight, or more primary imagingdevices. A UAV may have one, two, three, four, five, six, seven, eight,or more secondary imaging devices. The UAV may have more primary imagingdevices than secondary imaging devices, or vice-versa. Optionally, thenumber of primary imaging devices may be equal to the number ofsecondary imaging devices.

In some embodiments, a primary imaging device may be the imaging deviceoriented substantially along the direction of motion of the UAV. Theprimary imaging device can be considered to be oriented substantiallyalong the direction of motion when the field of view of the primaryimaging device overlaps with the direction of motion. In someembodiments, a primary imaging device may be considered to be orientedsubstantially along the direction of motion of the UAV when the field ofview or optical axis is aligned with the direction of motion. In someembodiments, a primary imaging device may be considered to be orientedsubstantially along the direction of motion of the UAV when the field ofview or optical axis of the imaging device is within or about 10°, 20°,30°, 40°, 50°, 60°, 70°, 80°, 90°, 100°, 110°, 120°, 130°, 140°, 150°,160°, 170°, or 180° of the direction of motion. The primary imagingdevice may change as the direction of motion of the UAV changes. Aprimary imaging device may no longer be oriented in the direction ofmotion of the UAV after the direction of motion of the UAV changes.

FIG. 9 illustrates a top view of a UAV 1200 in motion with a primaryimaging device 1202 and secondary imaging devices 1204, 1206, 1208, inaccordance with embodiments. The UAV 1200 includes a vehicle body 1210carrying the imaging devices 1202, 1204, 1206, 1208. Each of the imagingdevices 1202, 1204, 1206, 1208 is coupled to a different side of the UAV1200, and has a corresponding field of view 1212, 1214, 1216, 1218. TheUAV 1200 can be propelled by one or more propulsion units (not shown) tomove along a movement direction (indicated by the arrow 1220). Themovement direction of the UAV may be in any direction. The UAV may movehorizontally (e.g., forward, backward, left, right), vertically (e.g.,up, down) or in any combination. In the embodiment of FIG. 9, theimaging device 1202 is designated as the primary imaging device becausethe field of view 1212 is oriented along the movement direction, whilethe fields of view 1214, 1216, 1218 of the secondary imaging devices1204, 1206, 1208 are not oriented along the movement direction. In someembodiments, if the movement direction subsequently changes, thedesignations of the primary and secondary imaging devices may change,such that the imaging device oriented along the new movement directionis designated to be the new primary imaging device.

In alternative embodiments, the selection of the primary imaging deviceis performed independently of the direction of motion of the UAV. Forexample, primary and secondary imaging devices may be selected based onother criteria (e.g., user preference, feature point number, imagequality, power consumption, etc.). Accordingly, the primary imagingdevice may not be oriented substantially along the movement direction ofthe UAV.

An image sequence taken by a primary imaging device may be processed toassess the state information of the UAV, if the image quality of theprimary imaging device satisfies a certain criteria or passes a certainthreshold. The criteria may be whether the overall image sequence met aminimum threshold for image quality, whether each image in the sequencemet a minimum threshold for image quality, or whether a subset of theimages in the image sequence met a minimum threshold for image quality.In some embodiments, the “number of feature points” will beinterchangeable with “image quality” for purposes of determining thecriteria whether images taken by the primary imaging device will be usedfor processing of UAV state information. For example, the minimumthreshold for image quality may be a feature point number, such as noless than about 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 150, 200, 500or more feature points. If the image quality of the primary imagingdevice does not satisfy a certain criteria and/or pass a certainthreshold, the image quality of the secondary imaging devices may bedetermined. At least one of the secondary imaging devices can beselected for processing of state information based on one or morecriteria.

The selection of the imaging devices for processing of state informationwhen the primary imaging device did not take an image sequence ofsatisfactory quality may be based on one or more criteria. The criteriamay be whether the overall image sequence or each image in the imagesequence met a minimum threshold for image quality. The criteria may bean imaging device that had an image sequence with the highest averageimage quality. The criteria may be an imaging device with a highestmaximum image quality within the image sequence. The criteria may be animaging device with the highest minimum image quality within the imagesequence. It is to be understood that the criteria will change accordingto the number of imaging devices that are to be selected. For example,if selecting two image devices, the criteria may be two imaging deviceswith the first and second highest maximum image quality within the imagesequence. In some embodiments, the “number of feature points” will beinterchangeable with “image quality” for purposes of determining thecriteria for the selection of imaging devices whose image sequence willbe assessed to determine state information. In some embodiments,secondary imaging devices whose image sequence was used in theprocessing of state information can be designated as a new primaryimaging device. The previous primary imaging device that did not have asatisfactory image quality may become a secondary imaging device. Insome embodiments, the direction of motion of the UAV may not beassociated with the primary and secondary imaging devices.

FIG. 10 illustrates a method 1300 for assessing state information basedon primary and secondary imaging devices, in accordance withembodiments. The steps 1310-1350 in FIG. 10 can be performed by one ormore processors, some of which may be carried by the UAV. In someembodiments, the method 1300 is performed by a navigation system for theUAV. Furthermore, the steps may be performed automatically withoutrequiring user input. The steps 1310-1350 in FIG. 10 may be repeated ata desired rate. The rate may be less than or equal to about every 0.005seconds, 0.01 seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20seconds, 50 seconds, 100 seconds, 200 seconds, 500 seconds, 1000seconds, or 3600 seconds. In some embodiments, the rate may be aboutonce every 0.02 seconds to about once every 0.1 seconds.

In step 1310, a plurality of images can be captured by each imagingdevice of the plurality of imaging devices over a time interval. Thetime interval may be less than or equal to about 0.005 seconds, 0.01seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 50seconds, 100 seconds, 200 seconds, 500 seconds, 1000 seconds, or 3600seconds. In some embodiments, the time interval may be within a rangefrom about 0.02 seconds to about 0.1 seconds. An image sequence capturedduring the time interval can be comprised of about 1 image, 2 images, 5images, 10 images, 20 images, 50 images, 100 images, 200 images, 500images, or 1000 images. The imaging devices may be arranged on the UAVsuch that each of the plurality of images is captured from a differentfield of view, as previously discussed herein. Optionally, the imagingdevices may be arranged on the UAV such that each of the plurality ofimages is captured with a different field of view. The plurality ofimaging devices may include one or more primary imaging devices (e.g., asingle primary imaging device) and one or more secondary imagingdevices.

In step 1320, the image quality of the plurality of images from theprimary imaging device can be assessed (e.g., with the aid of aprocessor) to determine whether the image quality meets a predeterminedthreshold, as previously described. In some embodiments, the imagequality may be determined based on the total number of feature points inan image. If said image quality in the sequence of images meets apredetermined threshold, the images captured by the primary imagingdevices may be used to assess or determine the state information.Alternatively, other types of criteria can be used to assess the imagequality of the images from the primary imaging device. The criteria maybe whether the overall image sequence or each image in the sequence meta minimum threshold for image quality. In some embodiments, the “numberof feature points” will be interchangeable with “image quality” forpurposes of determining the criteria of whether images taken by theprimary imaging device will be used for processing of UAV stateinformation.

In step 1325, if the image quality meets the predetermined threshold (orsatisfies the criteria) as assessed in step 1320, the state informationof the UAV can be assessed with the aid of the processor using theplurality of images taken by the primary imaging device.

In step 1330, if the image quality in step 1320 did not meet thepredetermined threshold (or satisfy the criteria), the image quality ofthe plurality of images from the one or more secondary imaging devicescan be assessed with the aid of a processor. In some embodiments, theimage quality may be assessed or determined by the total number offeature points in an image, as previously described herein.

In step 1340, at least one of the one or more secondary imaging devicescan be selected based on the assessment of step 1330. The selection ofthe imaging devices may be based on one or more criteria. The criteriamay be whether the overall image sequence or each image in the sequencemet a minimum threshold for image quality. The criteria may be animaging device that had an image sequence with the highest average imagequality. The criteria may be an imaging device with a highest maximumimage quality within the image sequence. The criteria may be an imagingdevice with the highest minimum image quality within the image sequence.It is to be understood that the criteria will change according to thenumber of imaging devices that are to be selected. For example, ifselecting two image devices, the criteria may be two imaging deviceswith the first and second highest maximum image quality within the imagesequence. In some embodiments, the “number of feature points” will beinterchangeable with “image quality” for purposes of determining thecriteria for the selection of imaging devices whose image sequence willbe assessed to determine state information.

In some embodiments, if the image sequences taken by the secondaryimaging device(s) are of lower image quality than the primary imagingdevice(s) (e.g., none of the imaging devices captured an image sequencethat passed the predetermined threshold), the primary imaging device canbe selected for the purpose of assessing state information of the UAVdespite not having passed the predetermined criteria or threshold.

In step 1350, the state information of the UAV can be assessed with theaid of the processor, using the plurality of images from the secondaryimaging device(s) selected in step 1340. The state information can bedetermined, e.g., based on the image sequences captured by the selectedimaging devices as previously described herein. In some embodiments,state information of the UAV can be assessed with the aid of theprocessor using the plurality of images taken by the primary imagingdevice if the secondary imaging devices took an image sequence of lowerquality than the primary imaging device. In some embodiments, theresults of the state information assessment can vary based on the fieldof view of the imaging sequence used. Accordingly, the assessment of thestate information (which can be used for navigation) can take intoaccount any changes in the field of view of the selected imagingdevice(s).

In some embodiments, the imaging device(s) whose image sequence(s) wereused to assess the state information of the UAV can become or remain“primary imaging device(s).” A primary imaging device is an imagingdevice whose image sequence is always assessed to determine imagequality and whose image sequence is used to assess state information ofthe UAV if the image quality is deemed satisfactory. Imaging device(s)whose image sequence(s) were not used to assess the state information ofthe UAV may become or remain “secondary imaging device(s).” A secondaryimaging device is an imaging device whose image sequence is assessed forimage quality only if the image quality of the image sequence taken bythe primary imaging device is deemed unsatisfactory.

Optionally, the state information assessed in step 1350 can be used as abasis for outputting signals to cause the UAV to navigate within theenvironment. The signal can include control signals for the propulsionsystem (e.g., rotors) of the UAV for effecting movement of the vehicle.The signal can be generated based on user commands that are input into aremote terminal or other user device and subsequently transmitted to theUAV. Alternatively, the signal can be autonomously generated by the UAV(e.g., an automated onboard controller). In some instances, the signalcan be generated semi-autonomously with contributions from user input aswell as automated.

FIG. 11 illustrates exemplary feature point numbers for three imagesequences obtained by three secondary imaging devices, in accordancewith embodiments. Image sequence 1405 shows feature point numbers forten image frames taken by the secondary imaging device #1. Imagesequence 1410 shows feature point numbers for ten image frames taken bythe secondary imaging device #2. Image sequence 1415 shows feature pointnumbers for ten image frames taken by the secondary imaging device #3.The imaging devices #1, #2, and #3 can be capturing images withdifferent fields of view and/or with different optical axes, aspreviously described herein. In the embodiment of FIG. 11 the number offeature points in each image frame taken by a “primary imaging device”is less than a threshold T1 during a time interval t (not shown). Thethreshold T1 may be about 100 feature points. Therefore, the imagestaken by the primary imaging device may not necessarily be used toassess state information of the UAV and image quality of the imagestaken by the secondary imaging devices which have different fields ofviews with respect to each other and with respect to the primary imagingdevice can be assessed.

The number of feature points extracted from each one of the 10 imageframes taken by each secondary imaging device during time t (e.g., 2seconds) can be counted in order to determine if there is an imagesequence that is better suited for assessing state information. In FIG.11, the minimum number of feature points in 10 frames taken by secondaryimaging device #1 is 51 (shown in Frame 2), the minimum number offeature points in 10 frames taken by secondary imaging device #2 is 62(shown in Frame 6), and the minimum number of feature points in 10frames taken by secondary imaging device #3 is 49 (shown in Frame 1).The secondary imaging device that had taken an image frame with themaximum value among these minimum numbers can be selected (e.g.,secondary imaging device #2). The selected secondary imaging device canbecome a primary imaging device and/or be selected to be used fordetermining state information. In some embodiments, if the minimumnumber of feature points in the 10 frames taken by the selectedsecondary imaging device is larger than a second threshold T2, then asecondary imaging device that had been selected can become a primaryimaging device and/or be selected to be used for determining stateinformation. The threshold T2 may be about 120 feature points. In someembodiments, the state information of the UAV can be processed using animage sequence taken by the new primary imaging device.

Alternatively, the maximum value among the number of feature points in10 image frames taken by the secondary imaging devices can be comparedto select an imaging device. In FIG. 11, this is 72 for secondaryimaging device #1 (shown in Frame 4), 82 for secondary imaging device #2(shown in Frame 4) and 77 for secondary imaging device #3 (shown inFrame 10). The secondary imaging device that had taken an image framewith the maximum value among these maximum numbers can be selected (e.g.secondary imaging device #2). The selected secondary imaging device canbecome a primary imaging device and/or be selected to be used forassessing state information. In some embodiments, if the maximum numberof feature points in the 10 frames taken by the selected secondaryimaging device is larger than a second threshold T2, then a secondaryimaging device that had been selected can become a primary imagingdevice and/or be selected to be used for determining state information.In some embodiments, the state information of the UAV can be processedusing an image sequence taken by the new primary imaging device.

Alternatively, the average or total value of the feature points in 10image frames taken by the secondary imaging devices can be compared toselect an imaging device. In FIG. 11, this is 61.1 (611 total) forsecondary imaging device #1, 71.3 (713 total) for secondary imagingdevice #2, and 64.1 (641 total) for secondary imaging device #3. Thesecondary imaging device with the highest average or total value offeature points can be selected (e.g. secondary imaging device #2). Theselected secondary imaging device can become a primary imaging deviceand/or be selected to be used for determining state information. In someembodiments, if the average or total number of feature points in the 10frames taken by the selected secondary imaging device is larger than asecond threshold T2, then a secondary imaging device that had beenselected can become a primary imaging device and/or be selected to beused for assessing state information. In some embodiments, the stateinformation of the UAV can be processed using an image sequence taken bythe new primary imaging device.

A secondary imaging device that has captured an image sequence that hasthe high image quality value can be chosen to assess the stateinformation of the UAV when the primary imaging device does not capturean image sequence suitable to be used in determining state information.In some embodiments, other criteria described previously may be chosenas a basis for selecting an imaging device best suited for assessingstate information of the UAV. The approach described herein and shown inFIG. 11 can be used in combination with other methods (e.g., performedas part of step 1030-1050 of method 1000). For example, the stateinformation may be assessed and be used as a basis to output controlsignals to one or more propulsion units mounted on the vehicle foreffecting movement of the vehicle.

The systems, devices, and methods described herein can be applied to awide variety of movable objects. As previously mentioned, anydescription herein of an aerial vehicle may apply to and be used for anymovable object. A movable object of the present invention can beconfigured to move within any suitable environment, such as in air(e.g., a fixed-wing aircraft, a rotary-wing aircraft, or an aircrafthaving neither fixed wings nor rotary wings), in water (e.g., a ship ora submarine), on ground (e.g., a motor vehicle, such as a car, truck,bus, van, motorcycle; a movable structure or frame such as a stick,fishing pole; or a train), under the ground (e.g., a subway), in space(e.g., a spaceplane, a satellite, or a probe), or any combination ofthese environments. The movable object can be a vehicle, such as avehicle described elsewhere herein. In some embodiments, the movableobject can be mounted on a living subject, such as a human or an animal.Suitable animals can include avians, canines, felines, equines, bovines,ovines, porcines, delphines, rodents, or insects.

The movable object may be capable of moving freely within theenvironment with respect to six degrees of freedom (e.g., three degreesof freedom in translation and three degrees of freedom in rotation).Alternatively, the movement of the movable object can be constrainedwith respect to one or more degrees of freedom, such as by apredetermined path, track, or orientation. The movement can be actuatedby any suitable actuation mechanism, such as an engine or a motor. Theactuation mechanism of the movable object can be powered by any suitableenergy source, such as electrical energy, magnetic energy, solar energy,wind energy, gravitational energy, chemical energy, nuclear energy, orany suitable combination thereof. The movable object may beself-propelled via a propulsion system, as described elsewhere herein.The propulsion system may optionally run on an energy source, such aselectrical energy, magnetic energy, solar energy, wind energy,gravitational energy, chemical energy, nuclear energy, or any suitablecombination thereof. Alternatively, the movable object may be carried bya living being.

In some instances, the movable object can be a vehicle. Suitablevehicles may include water vehicles, aerial vehicles, space vehicles, orground vehicles. For example, aerial vehicles may be fixed-wing aircraft(e.g., airplane, gliders), rotary-wing aircraft (e.g., helicopters,rotorcraft), aircraft having both fixed wings and rotary wings, oraircraft having neither (e.g., blimps, hot air balloons). A vehicle canbe self-propelled, such as self-propelled through the air, on or inwater, in space, or on or under the ground. A self-propelled vehicle canutilize a propulsion system, such as a propulsion system including oneor more engines, motors, wheels, axles, magnets, rotors, propellers,blades, nozzles, or any suitable combination thereof. In some instances,the propulsion system can be used to enable the movable object to takeoff from a surface, land on a surface, maintain its current positionand/or orientation (e.g., hover), change orientation, and/or changeposition.

The movable object can be controlled remotely by a user or controlledlocally by an occupant within or on the movable object. In someembodiments, the movable object is an unmanned movable object, such as aUAV. An unmanned movable object, such as a UAV, may not have an occupantonboard the movable object. The movable object can be controlled by ahuman or an autonomous control system (e.g., a computer control system),or any suitable combination thereof. The movable object can be anautonomous or semi-autonomous robot, such as a robot configured with anartificial intelligence.

The movable object can have any suitable size and/or dimensions. In someembodiments, the movable object may be of a size and/or dimensions tohave a human occupant within or on the vehicle. Alternatively, themovable object may be of size and/or dimensions smaller than thatcapable of having a human occupant within or on the vehicle. The movableobject may be of a size and/or dimensions suitable for being lifted orcarried by a human. Alternatively, the movable object may be larger thana size and/or dimensions suitable for being lifted or carried by ahuman. In some instances, the movable object may have a maximumdimension (e.g., length, width, height, diameter, diagonal) of less thanor equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Themaximum dimension may be greater than or equal to about: 2 cm, 5 cm, 10cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance betweenshafts of opposite rotors of the movable object may be less than orequal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.Alternatively, the distance between shafts of opposite rotors may begreater than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m,or 10 m.

In some embodiments, the movable object may have a volume of less than100 cm×100 cm×100 cm, less than 50 cm×50 cm×30 cm, or less than 5 cm×5cm×3 cm. The total volume of the movable object may be less than orequal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40 cm³, 50cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³, 300 cm³,500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³, 1 m³, or10 m³. Conversely, the total volume of the movable object may be greaterthan or equal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40cm³, 50 cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³,300 cm³, 500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³,1 m³, or 10 m³.

In some embodiments, the movable object may have a footprint (which mayrefer to the lateral cross-sectional area encompassed by the movableobject) less than or equal to about: 32,000 cm², 20,000 cm², 10,000 cm²,1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm². Conversely, thefootprint may be greater than or equal to about: 32,000 cm², 20,000 cm²,10,000 cm², 1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm².

In some instances, the movable object may weigh no more than 1000 kg.The weight of the movable object may be less than or equal to about:1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg,8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg,or 0.01 kg. Conversely, the weight may be greater than or equal toabout: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1kg, 0.05 kg, or 0.01 kg.

In some embodiments, a movable object may be small relative to a loadcarried by the movable object. The load may include a payload and/or acarrier, as described in further detail below. In some examples, a ratioof a movable object weight to a load weight may be greater than, lessthan, or equal to about 1:1. In some instances, a ratio of a movableobject weight to a load weight may be greater than, less than, or equalto about 1:1. Optionally, a ratio of a carrier weight to a load weightmay be greater than, less than, or equal to about 1:1. When desired, theratio of an movable object weight to a load weight may be less than orequal to: 1:2, 1:3, 1:4, 1:5, 1:10, or even less. Conversely, the ratioof a movable object weight to a load weight can also be greater than orequal to: 2:1, 3:1, 4:1, 5:1, 10:1, or even greater.

In some embodiments, the movable object may have low energy consumption.For example, the movable object may use less than about: 5 W/h, 4 W/h, 3W/h, 2 W/h, 1 W/h, or less. In some instances, a carrier of the movableobject may have low energy consumption. For example, the carrier may useless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less. Optionally,a payload of the movable object may have low energy consumption, such asless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less.

FIG. 12 illustrates an unmanned aerial vehicle (UAV) 1500, in accordancewith embodiments of the present invention. The UAV may be an example ofa movable object as described herein. The UAV 1500 can include apropulsion system having four rotors 1502, 1504, 1506, and 1508. Anynumber of rotors may be provided (e.g., one, two, three, four, five,six, seven, eight, or more). The rotors, rotor assemblies, or otherpropulsion systems of the unmanned aerial vehicle may enable theunmanned aerial vehicle to hover/maintain position, change orientation,and/or change location. The distance between shafts of opposite rotorscan be any suitable length 1510. For example, the length 1510 can beless than or equal to 2 m, or less than equal to 5 m. In someembodiments, the length 1510 can be within a range from 40 cm to 1 m,from 10 cm to 2 m, or from 5 cm to 5 m. Any description herein of a UAVmay apply to a movable object, such as a movable object of a differenttype, and vice versa.

In some embodiments, the movable object can be configured to carry aload. The load can include one or more of passengers, cargo, equipment,instruments, and the like. The load can be provided within a housing.The housing may be separate from a housing of the movable object, or bepart of a housing for an movable object. Alternatively, the load can beprovided with a housing while the movable object does not have ahousing. Alternatively, portions of the load or the entire load can beprovided without a housing. The load can be rigidly fixed relative tothe movable object. Optionally, the load can be movable relative to themovable object (e.g., translatable or rotatable relative to the movableobject).

In some embodiments, the load includes a payload. The payload can beconfigured not to perform any operation or function. Alternatively, thepayload can be a payload configured to perform an operation or function,also known as a functional payload. For example, the payload can includeone or more sensors for surveying one or more targets. Any suitablesensor can be incorporated into the payload, such as an image capturedevice (e.g., a camera), an audio capture device (e.g., a parabolicmicrophone), an infrared imaging device, or an ultraviolet imagingdevice. The sensor can provide static sensing data (e.g., a photograph)or dynamic sensing data (e.g., a video). In some embodiments, the sensorprovides sensing data for the target of the payload. Alternatively or incombination, the payload can include one or more emitters for providingsignals to one or more targets. Any suitable emitter can be used, suchas an illumination source or a sound source. In some embodiments, thepayload includes one or more transceivers, such as for communicationwith a module remote from the movable object. Optionally, the payloadcan be configured to interact with the environment or a target. Forexample, the payload can include a tool, instrument, or mechanismcapable of manipulating objects, such as a robotic arm.

Optionally, the load may include a carrier. The carrier can be providedfor the payload and the payload can be coupled to the movable object viathe carrier, either directly (e.g., directly contacting the movableobject) or indirectly (e.g., not contacting the movable object).Conversely, the payload can be mounted on the movable object withoutrequiring a carrier. The payload can be integrally formed with thecarrier. Alternatively, the payload can be releasably coupled to thecarrier. In some embodiments, the payload can include one or morepayload elements, and one or more of the payload elements can be movablerelative to the movable object and/or the carrier, as described above.

The carrier can be integrally formed with the movable object.Alternatively, the carrier can be releasably coupled to the movableobject. The carrier can be coupled to the movable object directly orindirectly. The carrier can provide support to the payload (e.g., carryat least part of the weight of the payload). The carrier can include asuitable mounting structure (e.g., a gimbal platform) capable ofstabilizing and/or directing the movement of the payload. In someembodiments, the carrier can be adapted to control the state of thepayload (e.g., position and/or orientation) relative to the movableobject. For example, the carrier can be configured to move relative tothe movable object (e.g., with respect to one, two, or three degrees oftranslation and/or one, two, or three degrees of rotation) such that thepayload maintains its position and/or orientation relative to a suitablereference frame regardless of the movement of the movable object. Thereference frame can be a fixed reference frame (e.g., the surroundingenvironment). Alternatively, the reference frame can be a movingreference frame (e.g., the movable object, a payload target).

In some embodiments, the carrier can be configured to permit movement ofthe payload relative to the carrier and/or movable object. The movementcan be a translation with respect to up to three degrees of freedom(e.g., along one, two, or three axes) or a rotation with respect to upto three degrees of freedom (e.g., about one, two, or three axes), orany suitable combination thereof.

In some instances, the carrier can include a carrier frame assembly anda carrier actuation assembly. The carrier frame assembly can providestructural support to the payload. The carrier frame assembly caninclude individual carrier frame components, some of which can bemovable relative to one another. The carrier actuation assembly caninclude one or more actuators (e.g., motors) that actuate movement ofthe individual carrier frame components. The actuators can permit themovement of multiple carrier frame components simultaneously, or may beconfigured to permit the movement of a single carrier frame component ata time. The movement of the carrier frame components can produce acorresponding movement of the payload. For example, the carrieractuation assembly can actuate a rotation of one or more carrier framecomponents about one or more axes of rotation (e.g., roll axis, pitchaxis, or yaw axis). The rotation of the one or more carrier framecomponents can cause a payload to rotate about one or more axes ofrotation relative to the movable object. Alternatively or incombination, the carrier actuation assembly can actuate a translation ofone or more carrier frame components along one or more axes oftranslation, and thereby produce a translation of the payload along oneor more corresponding axes relative to the movable object.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, can be controlled by a terminal. Theterminal can be a remote control device at a location distant from themovable object, carrier, and/or payload. The terminal can be disposed onor affixed to a support platform. Alternatively, the terminal can be ahandheld or wearable device. For example, the terminal can include asmartphone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal can include auser interface, such as a keyboard, mouse, joystick, touchscreen, ordisplay. Any suitable user input can be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal can be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal can be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal can be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal can include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload.

The terminal can include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal can be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal can display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the positioning of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 13 illustrates a movable object 1600 including a carrier 1602 and apayload 1604, in accordance with embodiments. Although the movableobject 1600 is depicted as an aircraft, this depiction is not intendedto be limiting, and any suitable type of movable object can be used, aspreviously described herein. One of skill in the art would appreciatethat any of the embodiments described herein in the context of aircraftsystems can be applied to any suitable movable object (e.g., an UAV). Insome instances, the payload 1604 may be provided on the movable object1600 without requiring the carrier 1602. The movable object 1600 mayinclude propulsion mechanisms 1606, a sensing system 1608, and acommunication system 1610.

The propulsion mechanisms 1606 can include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. For example, the propulsion mechanisms 1606 maybe rotor assemblies or other rotary propulsion units, as disclosedelsewhere herein. The movable object may have one or more, two or more,three or more, or four or more propulsion mechanisms. The propulsionmechanisms may all be of the same type. Alternatively, one or morepropulsion mechanisms can be different types of propulsion mechanisms.The propulsion mechanisms 1606 can be mounted on the movable object 1600using any suitable means, such as a support element (e.g., a driveshaft) as described elsewhere herein. The propulsion mechanisms 1606 canbe mounted on any suitable portion of the movable object 1600, such onthe top, bottom, front, back, sides, or suitable combinations thereof.

In some embodiments, the propulsion mechanisms 1606 can enable themovable object 1600 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1600 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1606 can be operable to permit the movableobject 1600 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanisms 1600 may becontrolled independently of the other propulsion mechanisms.

Alternatively, the propulsion mechanisms 1600 can be configured to becontrolled simultaneously. For example, the movable object 1600 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1600. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally rotors may spin in acounterclockwise direction. For example, the number of clockwise rotorsmay be equal to the number of counterclockwise rotors. The rotation rateof each of the horizontally oriented rotors can be varied independentlyin order to control the lift and/or thrust produced by each rotor, andthereby adjust the spatial disposition, velocity, and/or acceleration ofthe movable object 1600 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation).

The sensing system 1608 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1600 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors caninclude global positioning system (GPS) sensors, motion sensors,inertial sensors, proximity sensors, or image sensors. The sensing dataprovided by the sensing system 1608 can be used to control the spatialdisposition, velocity, and/or orientation of the movable object 1600(e.g., using a suitable processing unit and/or control module, asdescribed below). Alternatively, the sensing system 1608 can be used toprovide data regarding the environment surrounding the movable object,such as weather conditions, proximity to potential obstacles, locationof geographical features, location of manmade structures, and the like.

The communication system 1610 enables communication with terminal 1612having a communication system 1614 via wireless signals 1616. Thecommunication systems 1610, 1614 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication, such that data can betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1600 transmitting data to theterminal 1612, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1610 to one or morereceivers of the communication system 1612, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata can be transmitted in both directions between the movable object1600 and the terminal 1612. The two-way communication can involvetransmitting data from one or more transmitters of the communicationsystem 1610 to one or more receivers of the communication system 1614,and vice-versa.

In some embodiments, the terminal 1612 can provide control data to oneor more of the movable object 1600, carrier 1602, and payload 1604 andreceive information from one or more of the movable object 1600, carrier1602, and payload 1604 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier and/orpayload. For example, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the propulsion mechanisms 1606), or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 1602).The control data from the terminal may result in control of the payload,such as control of the operation of a camera or other image capturingdevice (e.g., taking still or moving pictures, zooming in or out,turning on or off, switching imaging modes, change image resolution,changing focus, changing depth of field, changing exposure time,changing viewing angle or field of view). In some instances, thecommunications from the movable object, carrier and/or payload mayinclude information from one or more sensors (e.g., of the sensingsystem 1608 or of the payload 1604). The communications may includesensed information from one or more different types of sensors (e.g.,GPS sensors, motion sensors, inertial sensor, proximity sensors, orimage sensors). Such information may pertain to the position (e.g.,location, orientation), movement, or acceleration of the movable object,carrier and/or payload. Such information from a payload may include datacaptured by the payload or a sensed state of the payload. The controldata provided transmitted by the terminal 1612 can be configured tocontrol a state of one or more of the movable object 1600, carrier 1602,or payload 1604. Alternatively or in combination, the carrier 1602 andpayload 1604 can also each include a communication module configured tocommunicate with terminal 1612, such that the terminal can communicatewith and control each of the movable object 1600, carrier 1602, andpayload 1604 independently.

In some embodiments, the movable object 1600 can be configured tocommunicate with another remote device in addition to the terminal 1612,or instead of the terminal 1612. The terminal 1612 may also beconfigured to communicate with another remote device as well as themovable object 1600. For example, the movable object 1600 and/orterminal 1612 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicecan be configured to transmit data to the movable object 1600, receivedata from the movable object 1600, transmit data to the terminal 1612,and/or receive data from the terminal 1612. Optionally, the remotedevice can be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1600 and/orterminal 1612 can be uploaded to a website or server.

FIG. 14 is a schematic illustration by way of block diagram of a system1700 for controlling a movable object, in accordance with embodiments.The system 1700 can be used in combination with any suitable embodimentof the systems, devices, and methods disclosed herein. The system 1700can include a sensing module 1702, processing unit 1704, non-transitorycomputer readable medium 1706, control module 1708, and communicationmodule 1710.

The sensing module 1702 can utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors can includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1702 can beoperatively coupled to a processing unit 1704 having a plurality ofprocessors. In some embodiments, the sensing module can be operativelycoupled to a transmission module 1712 (e.g., a Wi-Fi image transmissionmodule) configured to directly transmit sensing data to a suitableexternal device or system. For example, the transmission module 1712 canbe used to transmit images captured by a camera of the sensing module1702 to a remote terminal.

The processing unit 1704 can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Theprocessing unit 1704 can be operatively coupled to a non-transitorycomputer readable medium 1706. The non-transitory computer readablemedium 1706 can store logic, code, and/or program instructionsexecutable by the processing unit 1704 for performing one or more steps.The non-transitory computer readable medium can include one or morememory units (e.g., removable media or external storage such as an SDcard or random access memory (RAM)). In some embodiments, data from thesensing module 1702 can be directly conveyed to and stored within thememory units of the non-transitory computer readable medium 1706. Thememory units of the non-transitory computer readable medium 1706 canstore logic, code and/or program instructions executable by theprocessing unit 1704 to perform any suitable embodiment of the methodsdescribed herein. For example, the processing unit 1704 can beconfigured to execute instructions causing one or more processors of theprocessing unit 1704 to analyze sensing data produced by the sensingmodule. The memory units can store sensing data from the sensing moduleto be processed by the processing unit 1704. In some embodiments, thememory units of the non-transitory computer readable medium 1706 can beused to store the processing results produced by the processing unit1704.

In some embodiments, the processing unit 1704 can be operatively coupledto a control module 1708 configured to control a state of the movableobject. For example, the control module 1708 can be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1708 can control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1704 can be operatively coupled to a communicationmodule 1710 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication can be used, such aswired communication or wireless communication. For example, thecommunication module 1710 can utilize one or more of local area networks(LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point(P2P) networks, telecommunication networks, cloud communication, and thelike. Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication module1710 can transmit and/or receive one or more of sensing data from thesensing module 1702, processing results produced by the processing unit1704, predetermined control data, user commands from a terminal orremote controller, and the like.

The components of the system 1700 can be arranged in any suitableconfiguration. For example, one or more of the components of the system1700 can be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 14 depicts asingle processing unit 1704 and a single non-transitory computerreadable medium 1706, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1700 can include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media can be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1700 can occur at one or more of theaforementioned locations.

As used herein A and/or B encompasses one or more of A or B, andcombinations thereof such as A and B.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

The invention claimed is:
 1. A navigation system, comprising: one ormore propulsion units configured to effect movement of a vehicle and aplurality of sensors each configured to capture a plurality of sensordata, wherein the plurality of sensors comprise a plurality of imagingdevices each configured to capture a plurality of images; and one ormore processors operably coupled to the plurality of sensors andindividually or collectively configured to: assess data quality of theplurality of sensor data from each of the plurality of sensors, whereinthe data quality is assessed based on a feature point number of eachimage captured by said plurality of imaging devices, wherein saidfeature point number of each image is calculated using a cornerdetection algorithm that is a Features from Accelerated Segmented Test(FAST) algorithm; select at least one of the plurality of sensors basedon the assessed data quality; and determine state information for thevehicle using the plurality of sensor data from the selected sensor(s),wherein the state information of the vehicle comprises information aboutan environment surrounding the vehicle.
 2. The system of claim 1,wherein the one or more processors are further configured to: outputcontrol signals to the one or more propulsion units for effecting themovement of the vehicle, based on the state information of the vehicle.3. The system of claim 1, wherein the plurality of sensors furthercomprise one or more of an ultrasonic sensor, a lidar sensor, or a radarsensor.
 4. The system of claim 1, wherein the plurality of imagingdevices are each oriented in a different direction relative to thevehicle.
 5. The system of claim 4, wherein the different directions areorthogonal directions.
 6. The system of claim 4, wherein the differentdirections comprise at least four different directions.
 7. The system ofclaim 1, wherein the data quality is based on a saliency of each imagecaptured by said plurality of imaging devices.
 8. The system of claim 1,wherein the data quality is based on at least one of an exposure levelor contrast level in each image captured by said plurality of imagingdevices.
 9. The system of claim 1, wherein the data quality is based ona suitability of said plurality of sensor data for use in determiningthe state information for the vehicle.
 10. The system of claim 1,wherein the one or more processors are configured to assess whether thedata quality of said plurality of sensor data exceeds a predeterminedthreshold.
 11. The system of claim 1, wherein the one or more processorsare configured to identify which of the plurality of sensor data has thehighest data quality.
 12. The system of claim 1, wherein the stateinformation further comprises at least one of a position, an attitude, avelocity, or an acceleration of the vehicle.
 13. The system of claim 1,wherein the one or more processors are configured to repeat steps toassess the data quality, select at least one of the plurality ofsensors, and determine state information during operation of thevehicle.
 14. The system of claim 1, wherein the one or more processorsare further configured to select at least one of the plurality ofsensors based on a power consumption of the plurality of sensors.
 15. Anavigation system, comprising: a plurality of imaging devices on avehicle, wherein each imaging device is configured to capture aplurality of images, wherein the plurality of imaging devices comprise aprimary imaging device and one or more secondary imaging devices; andone or more processors operably coupled to the plurality of imagingdevices and individually or collectively configured to: (1) assess imagequality of the plurality of images from the primary imaging device todetermine whether said image quality meets a predetermined threshold,and (2) assess image quality of the plurality of images from the one ormore secondary imaging devices if the image quality of the plurality ofimages from the primary imaging device does not meet the predeterminedthreshold, wherein the image quality of the plurality of images from theprimary and/or secondary imaging devices are each assessed based on afeature point number of each image of said plurality of images, whereinsaid feature point number of each image is calculated using a cornerdetection algorithm that is a Features from Accelerated Segmented Test(FAST) algorithm; select at least one of the one or more secondaryimaging devices based on the assessment of the image quality of theplurality of images from the one or more secondary imaging devices; anddetermine state information for the vehicle using the plurality ofimages from the selected secondary imaging device(s), wherein the stateinformation of the vehicle comprises information about an environmentsurrounding the vehicle.
 16. The system of claim 15, wherein the vehicleis an unmanned aerial vehicle.
 17. The system of claim 15, wherein theplurality of imaging devices are arranged on the vehicle such that eachimaging device of the plurality of imaging devices is configured tocapture the plurality of images from a different field of view.
 18. Thesystem of claim 15, wherein the plurality of images comprises aplurality of successive image frames captured over a predetermined timeinterval.
 19. The system of claim 18, wherein the predetermined timeinterval is within a range from about 0.02 seconds to about 0.1 seconds.20. The system of claim 15, wherein the image quality of the pluralityof images from the primary and/or secondary imaging devices are eachbased on saliency of each image of said plurality of images.
 21. Thesystem of claim 15, wherein the image quality of the plurality of imagesfrom the primary and/or secondary imaging devices are each based on atleast one of an exposure level or contrast level in each image of saidplurality of images.
 22. The system of claim 15, wherein the stateinformation further comprises at least one of a position, an attitude, avelocity, or an acceleration of the vehicle.
 23. The system of claim 22,wherein the attitude comprises at least one of a roll orientation, apitch orientation, or a yaw orientation of the vehicle.
 24. A method forcontrolling a moving vehicle operably coupled thereto a plurality ofimaging devices, comprising: capturing a plurality of images with eachimaging device of the plurality of imaging devices; and with aid of oneor more processors, individually or collectively, assessing imagequality of the plurality of images from each imaging device, wherein theimage quality is assessed based on a feature point number of each imagecaptured by said plurality of imaging devices, wherein said featurepoint number of each image is calculated using a corner detectionalgorithm that is a Features from Accelerated Segmented Test (FAST)algorithm; selecting at least one of the plurality of imaging devicesbased on the assessment of the image quality of the plurality of images;and determining state information for the vehicle using the plurality ofimages from the selected imaging device(s), wherein the stateinformation of the vehicle comprises information about an environmentsurrounding the vehicle.
 25. A method for assessing state information ofa moving vehicle attached thereto a plurality of imaging devices,comprising: capturing a plurality of images with each imaging device ofthe plurality of imaging devices, wherein the plurality of imagingdevices comprises a primary imaging device and one or more secondaryimaging devices; and with aid of one or more processors, individual orcollectively, (1) assessing image quality of the plurality of imagesfrom the primary imaging device to determine whether said image qualitymeets a predetermined threshold, and (2) assessing image quality of theplurality of images from the one or more secondary imaging device if theimage quality of the plurality of images from the primary imaging devicedoes not meet the predetermined threshold, wherein the image quality ofthe plurality of images from the primary and/or secondary imagingdevices are each assessed based on a feature point number of each imageof said plurality of images, wherein the feature point number of eachimage is calculated using a corner detection algorithm that is aFeatures from Accelerated Segmented Test (FAST) algorithm; selecting atleast one of the one or more secondary imaging devices based on theassessment of the image quality of the plurality of images from the oneor more secondary imaging devices; and determining state information forthe vehicle using the plurality of images from the selected secondaryimaging device(s), wherein the state information of the vehiclecomprises information about an environment surrounding the vehicle.